[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:30:55 dist-test-slave-fmv8 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:30:55 dist-test-slave-fmv8 krb5kdc[399](info): set up 2 sockets
Sep 29 18:30:55 dist-test-slave-fmv8 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:30:56.837154 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:30:56 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:30:56.959771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:30:57.066563 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:30:57.066812 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:30:57.066851 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:30:57.070691 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:30:57.070746 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:30:57.070765 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:30:57.070782 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:30:57.070797 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:30:57.075793 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:30:57.076738 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:30:57.077678 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:30:57.085959 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.086035 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.086036 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:57.086684 412 server_base.cc:1034] running on GCE node
I20260929 18:30:57.087319 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:30:57.088581 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:30:57.089784 412 hybrid_clock.cc:648] HybridClock initialized: now 1790706657089736 us; error 65 us; skew 500 ppm
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:30:57.092832 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:30:57.093667 412 webserver.cc:460] Webserver started at http://127.0.98.62:33221/ using document root <none> and password file <none>
I20260929 18:30:57.094219 412 fs_manager.cc:359] Metadata directory not provided
I20260929 18:30:57.094278 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:30:57.094453 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:30:57.096588 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "76dfffbf706d4ca7b8d92d34bc5892c8"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "bcea663a30b21e5a68db65510b1ffacb"
server_key_iv: "88c284a51bb017832e86849179f423e2"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.097102 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "76dfffbf706d4ca7b8d92d34bc5892c8"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "bcea663a30b21e5a68db65510b1ffacb"
server_key_iv: "88c284a51bb017832e86849179f423e2"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.100759 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:30:57.103211 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.104637 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:30:57.104741 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "76dfffbf706d4ca7b8d92d34bc5892c8"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "bcea663a30b21e5a68db65510b1ffacb"
server_key_iv: "88c284a51bb017832e86849179f423e2"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.104875 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:30:57.117910 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:30:57.121073 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:30:57.121237 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:30:57.128046 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32803
I20260929 18:30:57.128058 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32803 every 8 connection(s)
I20260929 18:30:57.129029 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:30:57.131799 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:30:57.136333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 412
I20260929 18:30:57.136499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:30:57.137318 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8: Bootstrap starting.
I20260929 18:30:57.137745 392 external_mini_cluster.cc:1428] Setting key 96c04c101a98347042f14f7b2135d0e1
I20260929 18:30:57.139814 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8: Neither blocks nor log segments found. Creating new log.
I20260929 18:30:57.140726 480 log.cc:824] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:30:57.142549 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8: No bootstrap required, opened a new log
I20260929 18:30:57.147055 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260929 18:30:57.147312 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:30:57.147361 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76dfffbf706d4ca7b8d92d34bc5892c8, State: Initialized, Role: FOLLOWER
I20260929 18:30:57.147805 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260929 18:30:57.147924 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:30:57.148011 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:30:57.148125 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:30:57.148986 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260929 18:30:57.149313 480 leader_election.cc:302] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76dfffbf706d4ca7b8d92d34bc5892c8; no voters:
I20260929 18:30:57.149588 480 leader_election.cc:288] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:30:57.149698 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:30:57.149963 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [term 1 LEADER]: Becoming Leader. State: Replica: 76dfffbf706d4ca7b8d92d34bc5892c8, State: Running, Role: LEADER
I20260929 18:30:57.150298 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } }
I20260929 18:30:57.150825 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:30:57.151420 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76dfffbf706d4ca7b8d92d34bc5892c8. Latest consensus state: current_term: 1 leader_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } }
I20260929 18:30:57.151531 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:30:57.151942 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:30:57.151949 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76dfffbf706d4ca7b8d92d34bc5892c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32803 } } }
I20260929 18:30:57.152205 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:30:57.155331 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:30:57.162190 492 catalog_manager.cc:1347] Generated new cluster ID: c0df042a5ac941bf90c826cd6259cbb8
I20260929 18:30:57.162249 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:30:57.165711 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.140566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.98.62:32803)
0929 18:30:57.140978 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.140997 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.142429 (+ 1432us) server_negotiation.cc:407] Connection header received
0929 18:30:57.146578 (+ 4149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.146599 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.146974 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.147325 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.150134 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.151563 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.153301 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.154002 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.154159 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.159361 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.159371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.159387 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.159420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.162341 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.162889 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.162894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.162898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.163002 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.163367 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.163370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.163372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.163736 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.163963 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.164565 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.164824 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":153,"threads_started":1}
I20260929 18:30:57.174085 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:30:57.174909 492 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:30:57.194607 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76dfffbf706d4ca7b8d92d34bc5892c8: Generated new TSK 0
I20260929 18:30:57.195343 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:30:57.223703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:30:57.334141 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:30:57.334434 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:30:57.334517 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:30:57.338599 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:30:57.338692 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:30:57.338783 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:30:57.344023 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:30:57.345094 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:30:57.346108 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:30:57.355700 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.355721 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.355700 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:57.355930 506 server_base.cc:1034] running on GCE node
I20260929 18:30:57.356379 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:30:57.357020 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:30:57.358186 506 hybrid_clock.cc:648] HybridClock initialized: now 1790706657358155 us; error 50 us; skew 500 ppm
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:30:57.360936 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:30:57.361778 506 webserver.cc:460] Webserver started at http://127.0.98.1:32877/ using document root <none> and password file <none>
I20260929 18:30:57.362306 506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:30:57.362396 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:30:57.362617 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:30:57.364636 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "387249683b424ccda3e4d292a60ce287"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "2b0a69b093b1f47857155071a9c070a0"
server_key_iv: "8bd96780eda9f2a047478e7216ecb436"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.365217 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "387249683b424ccda3e4d292a60ce287"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "2b0a69b093b1f47857155071a9c070a0"
server_key_iv: "8bd96780eda9f2a047478e7216ecb436"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.368852 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:30:57.371678 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.373139 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 18:30:57.373248 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "387249683b424ccda3e4d292a60ce287"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "2b0a69b093b1f47857155071a9c070a0"
server_key_iv: "8bd96780eda9f2a047478e7216ecb436"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.373377 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:30:57.399464 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:30:57.402725 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:30:57.402935 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:30:57.403527 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:30:57.404418 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:30:57.404486 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.404552 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:30:57.404599 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.413619 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42967
I20260929 18:30:57.413650 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42967 every 8 connection(s)
I20260929 18:30:57.414494 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:30:57.420241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 506
I20260929 18:30:57.420359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:30:57.420655 392 external_mini_cluster.cc:1428] Setting key 0120439ab99bde527d3f7a5b83ea5a8a
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:30:57.429199 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.416362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37699 (local address 127.0.98.62:32803)
0929 18:30:57.416505 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.416510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.417295 (+ 785us) server_negotiation.cc:407] Connection header received
0929 18:30:57.418567 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.418570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.418614 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.418689 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.419802 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.420473 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.421948 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.422215 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.422300 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.425092 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.425098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.425105 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.425137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.426890 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.427405 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.427412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.427416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.427510 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.427835 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.427839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.427840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.428090 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.428235 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.428863 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.429003 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:30:57.429925 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.416725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.1:37699)
0929 18:30:57.417151 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:30:57.417191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:30:57.418379 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:30:57.418884 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:30:57.418891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:30:57.419217 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:30:57.419674 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.419687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.420600 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.420604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.421821 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.421834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.422364 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.422368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.422588 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.423194 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:30:57.423219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:30:57.424929 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:30:57.427026 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.427031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.427042 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.427297 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.427609 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.427611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.427613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.427742 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.428351 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:30:57.428359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:30:57.428606 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:30:57.428803 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:30:57.429051 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:30:57.431221 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260929 18:30:57.431463 635 heartbeater.cc:464] Registering TS with master...
I20260929 18:30:57.431946 635 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:30:57.433799 444 ts_manager.cc:194] Registered new tserver with Master: 387249683b424ccda3e4d292a60ce287 (127.0.98.1:42967)
I20260929 18:30:57.435045 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:30:57.447060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:30:57.553216 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:30:57.553447 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:30:57.553488 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:30:57.557762 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:30:57.557821 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:30:57.557938 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:30:57.563185 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:30:57.564136 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:30:57.565117 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:30:57.574383 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.574404 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:57.574533 640 server_base.cc:1034] running on GCE node
W20260929 18:30:57.574394 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:57.575299 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:30:57.575946 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:30:57.577175 640 hybrid_clock.cc:648] HybridClock initialized: now 1790706657577157 us; error 29 us; skew 500 ppm
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:30:57.579873 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:30:57.580724 640 webserver.cc:460] Webserver started at http://127.0.98.2:37223/ using document root <none> and password file <none>
I20260929 18:30:57.581256 640 fs_manager.cc:359] Metadata directory not provided
I20260929 18:30:57.581348 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:30:57.581560 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:30:57.583877 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "16b89f649f6844da86f735f4818888e9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "dd4862f5fe9ca5f401210615ec56504c"
server_key_iv: "a6ac2334a26a4f1ea4312075666e2fed"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.584440 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "16b89f649f6844da86f735f4818888e9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "dd4862f5fe9ca5f401210615ec56504c"
server_key_iv: "a6ac2334a26a4f1ea4312075666e2fed"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.587813 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:30:57.590220 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.591369 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:30:57.591502 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "16b89f649f6844da86f735f4818888e9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "dd4862f5fe9ca5f401210615ec56504c"
server_key_iv: "a6ac2334a26a4f1ea4312075666e2fed"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.591610 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:30:57.613992 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:30:57.617383 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:30:57.617595 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:30:57.618189 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:30:57.618985 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:30:57.619062 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.619132 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:30:57.619184 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.628731 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36619
I20260929 18:30:57.628695 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36619 every 8 connection(s)
I20260929 18:30:57.629628 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:30:57.632725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 640
I20260929 18:30:57.632848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:30:57.633147 392 external_mini_cluster.cc:1428] Setting key f76248dfd4b68fde2b0b2c3fc67c7a66
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:30:57.645327 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.631588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40183 (local address 127.0.98.62:32803)
0929 18:30:57.631726 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.631731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.632647 (+ 916us) server_negotiation.cc:407] Connection header received
0929 18:30:57.634257 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.634260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.634302 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.634391 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.636072 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.636964 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.638099 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.638510 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.638630 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.641150 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.641155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.641158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.641185 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.642912 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.643444 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.643447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.643448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.643523 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.643838 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.643840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.643842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.644010 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.644266 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.644773 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.645085 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:30:57.645898 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.631971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.2:40183)
0929 18:30:57.632447 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 18:30:57.632525 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:30:57.634063 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:30:57.634648 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:30:57.634655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:30:57.635174 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:30:57.635909 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.635932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.637098 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.637100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.637972 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.637980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.638691 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.638695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.638869 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.639509 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:30:57.639524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:30:57.640980 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:30:57.643034 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.643038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.643050 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.643311 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.643612 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.643614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.643616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.643744 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.644278 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:30:57.644287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:30:57.644543 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 18:30:57.644703 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:30:57.644979 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":148,"threads_started":1}
I20260929 18:30:57.647224 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260929 18:30:57.647480 769 heartbeater.cc:464] Registering TS with master...
I20260929 18:30:57.647957 769 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260929 18:30:57.649086 444 ts_manager.cc:194] Registered new tserver with Master: 16b89f649f6844da86f735f4818888e9 (127.0.98.2:36619)
I20260929 18:30:57.650053 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:30:57.661103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:30:57.777433 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:30:57.777719 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:30:57.777807 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:30:57.782148 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:30:57.782224 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:30:57.782361 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:30:57.788635 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32803
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:30:57.789716 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:30:57.790768 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:30:57.800818 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.800834 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:57.800881 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:57.801067 774 server_base.cc:1034] running on GCE node
I20260929 18:30:57.801517 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:30:57.802109 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:30:57.803313 774 hybrid_clock.cc:648] HybridClock initialized: now 1790706657803294 us; error 33 us; skew 500 ppm
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:30:57.806071 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:30:57.806888 774 webserver.cc:460] Webserver started at http://127.0.98.3:42865/ using document root <none> and password file <none>
I20260929 18:30:57.807432 774 fs_manager.cc:359] Metadata directory not provided
I20260929 18:30:57.807490 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:30:57.807690 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:30:57.809864 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "47ca554457e94469a79598cda1fb8ce9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "1c9139ee6fefe0428ed79383edcd8602"
server_key_iv: "f2a9bee2ac5d1bf007ab0e5d348d29c8"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.810462 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47ca554457e94469a79598cda1fb8ce9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "1c9139ee6fefe0428ed79383edcd8602"
server_key_iv: "f2a9bee2ac5d1bf007ab0e5d348d29c8"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.814118 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:30:57.816468 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.817689 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:30:57.817782 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "47ca554457e94469a79598cda1fb8ce9"
format_stamp: "Formatted at 2026-09-29 18:30:57 on dist-test-slave-fmv8"
server_key: "1c9139ee6fefe0428ed79383edcd8602"
server_key_iv: "f2a9bee2ac5d1bf007ab0e5d348d29c8"
server_key_version: "encryptionkey@0"
I20260929 18:30:57.817929 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:30:57.846103 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:30:57.849373 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:30:57.849573 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:30:57.850169 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:30:57.850977 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:30:57.851047 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.851117 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:30:57.851168 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:57.861599 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38009
I20260929 18:30:57.861625 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38009 every 8 connection(s)
I20260929 18:30:57.862470 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:30:57.868124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 774
I20260929 18:30:57.868257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:30:57.868501 392 external_mini_cluster.cc:1428] Setting key 36bb13c445c5ca68a4fdb9a9c7e7ac28
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:30:57.877041 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.864232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34061 (local address 127.0.98.62:32803)
0929 18:30:57.864405 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.864408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.865219 (+ 811us) server_negotiation.cc:407] Connection header received
0929 18:30:57.866502 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.866505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.866550 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.866628 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.867760 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.868614 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.869977 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.870287 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.870371 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.873352 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.873355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.873357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.873381 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.874837 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.875320 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.875323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.875325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.875399 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.875719 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.875721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.875722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.875934 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.876016 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.876654 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.876761 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:30:57.877766 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.864601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32803 (local address 127.0.98.3:34061)
0929 18:30:57.865067 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 18:30:57.865109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:30:57.866286 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:30:57.866820 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:30:57.866827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:30:57.867174 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:30:57.867632 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.867644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.868730 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.868733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.869835 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:30:57.869844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.870464 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.870468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:30:57.870662 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.871464 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:30:57.871480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:30:57.873038 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:30:57.874947 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.874952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.874968 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.875228 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.875495 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:30:57.875498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:30:57.875500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:30:57.875633 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:30:57.876062 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:30:57.876067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:30:57.876392 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 18:30:57.876579 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 18:30:57.876837 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
I20260929 18:30:57.879014 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32803
I20260929 18:30:57.879258 903 heartbeater.cc:464] Registering TS with master...
I20260929 18:30:57.879733 903 heartbeater.cc:511] Master 127.0.98.62:32803 requested a full tablet report, sending...
I20260929 18:30:57.880796 444 ts_manager.cc:194] Registered new tserver with Master: 47ca554457e94469a79598cda1fb8ce9 (127.0.98.3:38009)
I20260929 18:30:57.881363 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34061
I20260929 18:30:57.883214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:30:57.910277 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.900274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.0.98.1:42967)
0929 18:30:57.900542 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.900561 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.900588 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:30:57.900782 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.900789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.900987 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.901138 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.901829 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.902649 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.903514 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.903906 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.904031 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.906593 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.906602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.906620 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.906645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.908376 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.908830 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.908836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.908837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.908922 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.909221 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.909225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.909226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.909522 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.909752 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.909934 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.910092 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":60,"threads_started":1}
W20260929 18:30:57.911383 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52802
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:30:57 dist-test-slave-fmv8 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:30:57.938874 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:57.930204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.0.98.1:42967)
0929 18:30:57.930360 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:57.930363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:57.930586 (+ 223us) server_negotiation.cc:407] Connection header received
0929 18:30:57.930752 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:57.930754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:57.930864 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:57.930933 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:57.931630 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.932251 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.933160 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:57.933404 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:57.933493 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:57.935802 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:57.935805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:57.935807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:57.935832 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:57.937276 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.937663 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.937667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.937668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.937756 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:57.938054 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:57.938057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:57.938058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:57.938222 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:57.938322 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:57.938644 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 18:30:57.938755 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:30:57.940002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 506
I20260929 18:30:57.945637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 640
I20260929 18:30:57.953377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 774
I20260929 18:30:57.958420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 412
2026-09-29T18:30:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2177 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:30:58 dist-test-slave-fmv8 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:30:58 dist-test-slave-fmv8 krb5kdc[917](info): set up 2 sockets
Sep 29 18:30:58 dist-test-slave-fmv8 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 18:30:58.987769 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:30:59 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:30:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:30:59.108013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:30:59.213874 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:30:59.214133 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:30:59.214174 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:30:59.218017 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:30:59.218065 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:30:59.218083 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:30:59.218099 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:30:59.218114 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:30:59.223109 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:30:59.224027 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:30:59.224997 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:30:59.233218 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:59.233294 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:30:59.233227 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:30:59.233429 930 server_base.cc:1034] running on GCE node
I20260929 18:30:59.234179 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:30:59.235205 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:30:59.236402 930 hybrid_clock.cc:648] HybridClock initialized: now 1790706659236382 us; error 30 us; skew 500 ppm
Sep 29 18:30:59 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:30:59.239164 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:30:59.239997 930 webserver.cc:460] Webserver started at http://127.0.98.62:34281/ using document root <none> and password file <none>
I20260929 18:30:59.240594 930 fs_manager.cc:359] Metadata directory not provided
I20260929 18:30:59.240686 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:30:59.240913 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:30:59.242828 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "03ee758c973741ef9672b58bca345f91"
format_stamp: "Formatted at 2026-09-29 18:30:59 on dist-test-slave-fmv8"
server_key: "a56a729421d4fb2c32df1acb96345527"
server_key_iv: "d20f7c51969ae38a359dcd19266f1c66"
server_key_version: "encryptionkey@0"
I20260929 18:30:59.243423 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "03ee758c973741ef9672b58bca345f91"
format_stamp: "Formatted at 2026-09-29 18:30:59 on dist-test-slave-fmv8"
server_key: "a56a729421d4fb2c32df1acb96345527"
server_key_iv: "d20f7c51969ae38a359dcd19266f1c66"
server_key_version: "encryptionkey@0"
I20260929 18:30:59.246956 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:30:59.249511 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:30:59.250813 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:30:59.250970 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "03ee758c973741ef9672b58bca345f91"
format_stamp: "Formatted at 2026-09-29 18:30:59 on dist-test-slave-fmv8"
server_key: "a56a729421d4fb2c32df1acb96345527"
server_key_iv: "d20f7c51969ae38a359dcd19266f1c66"
server_key_version: "encryptionkey@0"
I20260929 18:30:59.251084 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:30:59.281250 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:30:59.284425 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:30:59.284674 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:30:59.291839 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43983
I20260929 18:30:59.291864 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43983 every 8 connection(s)
I20260929 18:30:59.292870 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:30:59.294144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 930
I20260929 18:30:59.294237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:30:59.294499 392 external_mini_cluster.cc:1428] Setting key 8f4058be0bfed10618f530e1bc1e7f0d
I20260929 18:30:59.295969 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:30:59.302337 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91: Bootstrap starting.
Sep 29 18:30:59 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:30:59.305079 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91: Neither blocks nor log segments found. Creating new log.
I20260929 18:30:59.305967 998 log.cc:824] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91: Log is configured to *not* fsync() on all Append() calls
I20260929 18:30:59.308195 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91: No bootstrap required, opened a new log
I20260929 18:30:59.309902 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:30:59.295906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.98.62:43983)
0929 18:30:59.296370 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 18:30:59.296384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:30:59.296425 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:30:59.297075 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:30:59.297090 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:30:59.297411 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:30:59.297688 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:30:59.298568 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:59.299581 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:59.300811 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:30:59.301284 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:30:59.301402 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:30:59.304480 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:30:59.304486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:30:59.304503 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:30:59.304534 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:30:59.306608 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:59.307025 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:59.307031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:59.307036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:59.307168 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:30:59.307437 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:30:59.307441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:30:59.307442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:30:59.307929 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:30:59.308118 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:30:59.308446 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 18:30:59.308747 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":174,"threads_started":1}
I20260929 18:30:59.311877 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } }
I20260929 18:30:59.312179 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:30:59.312275 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ee758c973741ef9672b58bca345f91, State: Initialized, Role: FOLLOWER
I20260929 18:30:59.312680 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } }
I20260929 18:30:59.312809 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:30:59.312892 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:30:59.312987 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:30:59.313956 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } }
I20260929 18:30:59.314337 998 leader_election.cc:302] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03ee758c973741ef9672b58bca345f91; no voters:
I20260929 18:30:59.314669 998 leader_election.cc:288] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:30:59.314828 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:30:59.315053 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [term 1 LEADER]: Becoming Leader. State: Replica: 03ee758c973741ef9672b58bca345f91, State: Running, Role: LEADER
I20260929 18:30:59.315356 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } }
I20260929 18:30:59.315842 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:30:59.317034 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03ee758c973741ef9672b58bca345f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } } }
I20260929 18:30:59.317049 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ee758c973741ef9672b58bca345f91. Latest consensus state: current_term: 1 leader_uuid: "03ee758c973741ef9672b58bca345f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ee758c973741ef9672b58bca345f91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43983 } } }
I20260929 18:30:59.317230 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [sys.catalog]: This master's current role is: LEADER
I20260929 18:30:59.317508 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91 [sys.catalog]: This master's current role is: LEADER
I20260929 18:30:59.318495 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:30:59.321322 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:30:59.326228 1013 catalog_manager.cc:1347] Generated new cluster ID: 12ef7cb1e91e4392a78956923747f6e1
I20260929 18:30:59.326293 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:30:59.342332 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:30:59.343367 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:30:59.351684 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03ee758c973741ef9672b58bca345f91: Generated new TSK 0
I20260929 18:30:59.352563 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260929 18:31:00.306799 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.944s user 0.000s sys 0.003s
I20260929 18:31:00.307494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:00.424651 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:00.424892 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:00.424935 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:00.428975 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:00.429033 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:00.429123 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:00.435158 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:00.436328 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:00.437353 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:00.447224 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:00.447212 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:00.447322 1024 server_base.cc:1034] running on GCE node
W20260929 18:31:00.447212 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:00.447901 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:00.448544 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:00.449740 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790706660449701 us; error 52 us; skew 500 ppm
Sep 29 18:31:00 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:00.457089 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:00.457993 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41935/ using document root <none> and password file <none>
I20260929 18:31:00.458578 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:00.458647 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:00.458811 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:00.460949 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8c83dac6e2e4d968afd32ffd2afd300"
format_stamp: "Formatted at 2026-09-29 18:31:00 on dist-test-slave-fmv8"
server_key: "30f6c2fa0a9e44f7b37726049e9c3272"
server_key_iv: "629beaee5d469c0d15408bc76218057a"
server_key_version: "encryptionkey@0"
I20260929 18:31:00.461454 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8c83dac6e2e4d968afd32ffd2afd300"
format_stamp: "Formatted at 2026-09-29 18:31:00 on dist-test-slave-fmv8"
server_key: "30f6c2fa0a9e44f7b37726049e9c3272"
server_key_iv: "629beaee5d469c0d15408bc76218057a"
server_key_version: "encryptionkey@0"
I20260929 18:31:00.465312 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:31:00.467660 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:00.469028 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:00.469164 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "a8c83dac6e2e4d968afd32ffd2afd300"
format_stamp: "Formatted at 2026-09-29 18:31:00 on dist-test-slave-fmv8"
server_key: "30f6c2fa0a9e44f7b37726049e9c3272"
server_key_iv: "629beaee5d469c0d15408bc76218057a"
server_key_version: "encryptionkey@0"
I20260929 18:31:00.469269 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:00.479552 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:00.482645 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:00.482846 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:00.483429 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:00.484277 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:00.484323 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:00.484385 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:00.484413 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:00.492594 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46715
I20260929 18:31:00.492661 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46715 every 8 connection(s)
I20260929 18:31:00.493505 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:00.494547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1024
I20260929 18:31:00.494721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:00.494975 392 external_mini_cluster.cc:1428] Setting key 1adce8d020b46edd995d0c2eb4b61858
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 18:31:00 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:00.615607 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:00.495461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37545 (local address 127.0.98.62:43983)
0929 18:31:00.495762 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:00.495766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:00.496699 (+ 933us) server_negotiation.cc:407] Connection header received
0929 18:31:00.498025 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:00.498028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:00.498094 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:00.498181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:00.499620 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:00.500198 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:00.502495 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:00.502978 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:00.503106 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:00.611228 (+108122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:00.611232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:00.611235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:00.611274 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:00.613204 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:00.613846 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:00.613850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:00.613852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:00.613946 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:00.614358 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:00.614362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:00.614364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:00.614580 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:00.614691 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:00.615288 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 18:31:00.615424 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":135,"threads_started":1}
I20260929 18:31:00.616734 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:00.495995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43983 (local address 127.0.98.1:37545)
0929 18:31:00.496559 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:00.496607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:00.497821 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:00.498345 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:00.498351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:00.498706 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:00.499375 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:00.499397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:00.500348 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:00.500355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:00.501929 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:00.501944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:00.503165 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:00.503169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:00.503384 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:00.504134 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:00.504182 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:00.610905 (+106723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:00.613366 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:00.613373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:00.613393 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:00.613723 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:00.614079 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:00.614083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:00.614085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:00.614255 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:00.614691 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:00.614699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:00.615004 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 18:31:00.615250 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 18:31:00.615688 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":197,"threads_started":1}
I20260929 18:31:00.618263 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43983
I20260929 18:31:00.618528 1153 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:00.619095 1153 heartbeater.cc:511] Master 127.0.98.62:43983 requested a full tablet report, sending...
I20260929 18:31:00.620754 962 ts_manager.cc:194] Registered new tserver with Master: a8c83dac6e2e4d968afd32ffd2afd300 (127.0.98.1:46715)
I20260929 18:31:00.621944 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37545
I20260929 18:31:01.624676 1153 heartbeater.cc:503] Master 127.0.98.62:43983 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260929 18:31:01.680800 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.184s user 0.003s sys 0.000s
I20260929 18:31:01.681337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:01.793363 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:01.793650 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:01.793695 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:01.797899 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:01.797961 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:01.798020 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:01.803198 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:01.804198 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:01.805177 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:01.814893 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:01.814921 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:01.814896 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:01.815676 1159 server_base.cc:1034] running on GCE node
I20260929 18:31:01.816129 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:01.816980 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:01.818167 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790706661818150 us; error 42 us; skew 500 ppm
Sep 29 18:31:01 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:01.821014 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:01.821920 1159 webserver.cc:460] Webserver started at http://127.0.98.2:43503/ using document root <none> and password file <none>
I20260929 18:31:01.822495 1159 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:01.822587 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:01.822805 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:01.824879 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "030e2d6f3abc4ba892d57c0f9b7519db"
format_stamp: "Formatted at 2026-09-29 18:31:01 on dist-test-slave-fmv8"
server_key: "740af0f9ece0c8e0762c8e50e9666ad6"
server_key_iv: "fae1cb0570afed568a50679750079748"
server_key_version: "encryptionkey@0"
I20260929 18:31:01.825433 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "030e2d6f3abc4ba892d57c0f9b7519db"
format_stamp: "Formatted at 2026-09-29 18:31:01 on dist-test-slave-fmv8"
server_key: "740af0f9ece0c8e0762c8e50e9666ad6"
server_key_iv: "fae1cb0570afed568a50679750079748"
server_key_version: "encryptionkey@0"
I20260929 18:31:01.829104 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:01.831542 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:01.832809 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:01.832940 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "030e2d6f3abc4ba892d57c0f9b7519db"
format_stamp: "Formatted at 2026-09-29 18:31:01 on dist-test-slave-fmv8"
server_key: "740af0f9ece0c8e0762c8e50e9666ad6"
server_key_iv: "fae1cb0570afed568a50679750079748"
server_key_version: "encryptionkey@0"
I20260929 18:31:01.833052 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:01.856267 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:01.859429 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:01.859664 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:01.860347 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:01.861172 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:01.861217 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:01.861289 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:01.861328 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:01.870525 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46805
I20260929 18:31:01.870625 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46805 every 8 connection(s)
I20260929 18:31:01.871507 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:01.877967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1159
I20260929 18:31:01.878141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:01.878536 392 external_mini_cluster.cc:1428] Setting key 5e20dad3c6cae2ca5c06a47ac34c40fc
Sep 29 18:31:01 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:01.887324 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:01.873652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46451 (local address 127.0.98.62:43983)
0929 18:31:01.873919 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:01.873923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:01.874795 (+ 872us) server_negotiation.cc:407] Connection header received
0929 18:31:01.876127 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:01.876131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:01.876267 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:01.876360 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:01.877591 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:01.878234 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:01.879741 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:01.880016 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:01.880133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:01.883383 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:01.883387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:01.883389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:01.883416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:01.885053 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:01.885685 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:01.885691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:01.885693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:01.885782 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:01.886156 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:01.886160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:01.886162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:01.886352 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:01.886466 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:01.887002 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 18:31:01.887134 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
I20260929 18:31:01.888257 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:01.874126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43983 (local address 127.0.98.2:46451)
0929 18:31:01.874630 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:01.874675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:01.875903 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:01.876535 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:01.876541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:01.876900 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:01.877389 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:01.877402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:01.878392 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:01.878396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:01.879617 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:01.879628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:01.880232 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:01.880238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:01.880433 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:01.881074 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:01.881100 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:01.883166 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:01.885185 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:01.885191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:01.885208 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:01.885572 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:01.885892 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:01.885895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:01.885897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:01.886055 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:01.886463 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:01.886470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:01.886738 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 18:31:01.886963 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 18:31:01.887324 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":154,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:01.889547 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43983
I20260929 18:31:01.889803 1288 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:01.890291 1288 heartbeater.cc:511] Master 127.0.98.62:43983 requested a full tablet report, sending...
I20260929 18:31:01.891320 962 ts_manager.cc:194] Registered new tserver with Master: 030e2d6f3abc4ba892d57c0f9b7519db (127.0.98.2:46805)
I20260929 18:31:01.891970 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 18:31:02.670595 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.790s user 0.000s sys 0.004s
I20260929 18:31:02.671132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:02.790459 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:02.790714 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:02.790757 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:02.795017 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:02.795082 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:02.795141 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:02.800478 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:02.801478 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:02.802500 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:02.812135 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:02.812155 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:02.812189 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:02.812577 1294 server_base.cc:1034] running on GCE node
I20260929 18:31:02.812994 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:02.813635 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:02.814839 1294 hybrid_clock.cc:648] HybridClock initialized: now 1790706662814825 us; error 29 us; skew 500 ppm
Sep 29 18:31:02 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:02.818022 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:02.818866 1294 webserver.cc:460] Webserver started at http://127.0.98.3:42487/ using document root <none> and password file <none>
I20260929 18:31:02.819448 1294 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:02.819547 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:02.819779 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:02.821841 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "a15bd1a457204b3291a8c95b0c8c7039"
format_stamp: "Formatted at 2026-09-29 18:31:02 on dist-test-slave-fmv8"
server_key: "aeff40c8c25333ba57c3754aa5dda2b5"
server_key_iv: "7e16f45ea0b41b700eb1e25b1b617996"
server_key_version: "encryptionkey@0"
I20260929 18:31:02.822414 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a15bd1a457204b3291a8c95b0c8c7039"
format_stamp: "Formatted at 2026-09-29 18:31:02 on dist-test-slave-fmv8"
server_key: "aeff40c8c25333ba57c3754aa5dda2b5"
server_key_iv: "7e16f45ea0b41b700eb1e25b1b617996"
server_key_version: "encryptionkey@0"
I20260929 18:31:02.826062 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:31:02.828600 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:02.829898 1294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:31:02.830071 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "a15bd1a457204b3291a8c95b0c8c7039"
format_stamp: "Formatted at 2026-09-29 18:31:02 on dist-test-slave-fmv8"
server_key: "aeff40c8c25333ba57c3754aa5dda2b5"
server_key_iv: "7e16f45ea0b41b700eb1e25b1b617996"
server_key_version: "encryptionkey@0"
I20260929 18:31:02.830192 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:02.847487 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:02.850746 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:02.850982 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:02.851589 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:02.852528 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:02.852600 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:02.852664 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:02.852720 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:02.863044 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45093
I20260929 18:31:02.863091 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45093 every 8 connection(s)
I20260929 18:31:02.863996 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:02.868312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1294
I20260929 18:31:02.868448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:02.868711 392 external_mini_cluster.cc:1428] Setting key 84d56ae2e87919907de95f608ff7889f
Sep 29 18:31:02 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:02.881944 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.866043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45601 (local address 127.0.98.62:43983)
0929 18:31:02.866292 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:02.866295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:02.868780 (+ 2485us) server_negotiation.cc:407] Connection header received
0929 18:31:02.870977 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:02.870984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:02.871057 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:02.871130 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:02.872899 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.873822 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.875099 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.875319 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.875446 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.878200 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:02.878203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:02.878205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:02.878227 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:02.879639 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:02.880230 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:02.880233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:02.880235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:02.880311 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:02.880690 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:02.880693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:02.880694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:02.880865 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:02.880942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:02.881629 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 18:31:02.881752 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260929 18:31:02.882922 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.866465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43983 (local address 127.0.98.3:45601)
0929 18:31:02.868573 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:02.868639 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:02.870687 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:02.871599 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:02.871612 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:02.872098 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:02.872747 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.872763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.873949 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.873952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.874977 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.874987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.875597 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.875608 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.875751 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.876538 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:02.876554 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:02.878048 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:02.879797 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:02.879810 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:02.879823 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:02.880071 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:02.880429 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:02.880431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:02.880433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:02.880580 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:02.880993 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:02.880999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:02.881327 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 18:31:02.881567 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 18:31:02.881932 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":109,"threads_started":1}
I20260929 18:31:02.884076 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43983
I20260929 18:31:02.884397 1423 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:02.884902 1423 heartbeater.cc:511] Master 127.0.98.62:43983 requested a full tablet report, sending...
I20260929 18:31:02.886044 962 ts_manager.cc:194] Registered new tserver with Master: a15bd1a457204b3291a8c95b0c8c7039 (127.0.98.3:45093)
I20260929 18:31:02.886612 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45601
I20260929 18:31:02.894081 1288 heartbeater.cc:503] Master 127.0.98.62:43983 was elected leader, sending a full tablet report...
I20260929 18:31:02.894685 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:31:02.905787 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.898333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.98.62:43983)
0929 18:31:02.898465 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:02.898469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:02.898728 (+ 259us) server_negotiation.cc:407] Connection header received
0929 18:31:02.898795 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:02.898798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:02.898835 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:02.898915 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:02.899576 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.900099 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.900992 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.901226 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.901306 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.902620 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:02.902622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:02.902624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:02.902647 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:02.904241 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:02.904611 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:02.904614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:02.904615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:02.904685 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:02.904969 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:02.904972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:02.904973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:02.905192 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:02.905357 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:02.905544 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:31:02.905654 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:31:02.910898 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:02.914084 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:02.929337 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.923201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46715 (local address 127.0.0.1:53066)
0929 18:31:02.923906 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:02.923942 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:02.924104 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:02.924979 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:02.924987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:02.925013 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:02.925158 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.925166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.926668 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.926671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.927824 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.927834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.928913 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.928915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.929000 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.929032 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:31:02.929121 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:31:02.929213 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":130,"threads_started":1}
I20260929 18:31:02.929435 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.923378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.0.98.1:46715)
0929 18:31:02.924439 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:02.924447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:02.924471 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:02.924546 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:02.924550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:02.924780 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:02.924963 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:02.925312 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.926509 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.927968 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.928781 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.928902 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.929061 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:02.929167 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:31:02.929289 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":141,"threads_started":1}
I20260929 18:31:02.930289 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.923216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46805 (local address 127.0.0.1:47506)
0929 18:31:02.924004 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:02.924020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:02.924119 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:02.925908 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:02.925911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:02.925932 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:02.926011 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.926015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.927083 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.927090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.928581 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.928592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.929870 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.929873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.929943 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.929962 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:31:02.930065 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:31:02.930145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":36,"thread_start_us":167,"threads_started":1}
I20260929 18:31:02.930382 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.924507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.98.2:46805)
0929 18:31:02.924810 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:02.924817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:02.924845 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:31:02.924924 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:02.924929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:02.925608 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:02.925763 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:02.926136 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.926955 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.928723 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.929543 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.929663 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.929807 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:02.930125 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:31:02.930227 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260929 18:31:02.931239 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.924038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45093 (local address 127.0.0.1:41464)
0929 18:31:02.924651 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:02.924666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:02.924750 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:02.925672 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:02.925678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:02.925701 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:02.925810 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.925818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.927130 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.927133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.928581 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:02.928592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.930845 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.930849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:02.930944 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.930969 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:31:02.931038 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:31:02.931109 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":92,"thread_start_us":115,"threads_started":1}
I20260929 18:31:02.932673 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:02.924257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.3:45093)
0929 18:31:02.925122 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:02.925138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:02.925159 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:31:02.925231 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:02.925238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:02.925438 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:02.925580 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:02.925945 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.927009 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.929868 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:02.930699 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:02.930825 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:02.932416 (+ 1591us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:02.932475 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:31:02.932551 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":129,"threads_started":1}
I20260929 18:31:02.932802 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7a7a15bdf4d0d8febc995f1f52095 (DEFAULT_TABLE table=test-table [id=55ce5248c248417d926adeb7e629ae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:02.933470 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7a7a15bdf4d0d8febc995f1f52095 (DEFAULT_TABLE table=test-table [id=55ce5248c248417d926adeb7e629ae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:02.934070 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7a7a15bdf4d0d8febc995f1f52095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:02.934477 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7a7a15bdf4d0d8febc995f1f52095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:02.938659 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7a7a15bdf4d0d8febc995f1f52095 (DEFAULT_TABLE table=test-table [id=55ce5248c248417d926adeb7e629ae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:02.939529 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7a7a15bdf4d0d8febc995f1f52095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:02.941774 1442 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Bootstrap starting.
I20260929 18:31:02.943188 1443 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Bootstrap starting.
I20260929 18:31:02.944234 1442 tablet_bootstrap.cc:654] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:02.945024 1443 tablet_bootstrap.cc:654] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:02.945143 1442 log.cc:824] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:02.945833 1443 log.cc:824] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:02.947188 1445 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Bootstrap starting.
I20260929 18:31:02.947389 1442 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: No bootstrap required, opened a new log
I20260929 18:31:02.947588 1442 ts_tablet_manager.cc:1397] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:31:02.948637 1443 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: No bootstrap required, opened a new log
I20260929 18:31:02.948835 1443 ts_tablet_manager.cc:1397] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260929 18:31:02.949502 1445 tablet_bootstrap.cc:654] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:02.950996 1445 log.cc:824] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:02.951229 1442 raft_consensus.cc:348] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.951512 1442 raft_consensus.cc:374] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:02.951609 1442 raft_consensus.cc:729] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030e2d6f3abc4ba892d57c0f9b7519db, State: Initialized, Role: FOLLOWER
I20260929 18:31:02.952122 1442 consensus_queue.cc:260] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.953200 1442 ts_tablet_manager.cc:1428] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:31:02.953685 1443 raft_consensus.cc:348] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.953997 1443 raft_consensus.cc:374] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:02.954121 1443 raft_consensus.cc:729] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c83dac6e2e4d968afd32ffd2afd300, State: Initialized, Role: FOLLOWER
I20260929 18:31:02.954233 1445 tablet_bootstrap.cc:492] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: No bootstrap required, opened a new log
I20260929 18:31:02.954416 1445 ts_tablet_manager.cc:1397] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:31:02.954608 1443 consensus_queue.cc:260] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.955638 1443 ts_tablet_manager.cc:1428] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260929 18:31:02.958546 1445 raft_consensus.cc:348] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.958861 1445 raft_consensus.cc:374] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:02.958925 1445 raft_consensus.cc:729] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15bd1a457204b3291a8c95b0c8c7039, State: Initialized, Role: FOLLOWER
I20260929 18:31:02.959338 1445 consensus_queue.cc:260] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:02.960049 1423 heartbeater.cc:503] Master 127.0.98.62:43983 was elected leader, sending a full tablet report...
I20260929 18:31:02.960363 1445 ts_tablet_manager.cc:1428] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 18:31:02.996411 1154 tablet.cc:2369] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:03.115207 1424 tablet.cc:2369] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:03.123293 1289 tablet.cc:2369] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:03.126920 1450 raft_consensus.cc:482] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:03.127090 1450 raft_consensus.cc:504] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:03.128221 1450 leader_election.cc:288] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c83dac6e2e4d968afd32ffd2afd300 (127.0.98.1:46715), 030e2d6f3abc4ba892d57c0f9b7519db (127.0.98.2:46805)
I20260929 18:31:03.132064 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.128549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46715 (local address 127.0.98.3:50323)
0929 18:31:03.128665 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:03.128682 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:03.128791 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:03.129164 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:03.129168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:03.129185 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:03.129322 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:03.129328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.130185 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.130192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:03.130974 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:03.130982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.131641 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:03.131746 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.131767 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:31:03.131862 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:31:03.131929 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:31:03.132077 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.128804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50323 (local address 127.0.98.1:46715)
0929 18:31:03.128942 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:03.128946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:03.128958 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:31:03.129006 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:03.129008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:03.129047 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:03.129152 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:03.129443 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.130069 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131096 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.131522 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131639 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.131700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:03.131877 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:31:03.131945 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:31:03.132334 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.128698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42069 (local address 127.0.98.2:46805)
0929 18:31:03.128866 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:03.128871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:03.129308 (+ 437us) server_negotiation.cc:407] Connection header received
0929 18:31:03.129451 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:03.129454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:03.129506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:03.129590 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:03.129859 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.130349 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131372 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.131787 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.131960 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:03.132132 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:31:03.132227 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:31:03.132350 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.128549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46805 (local address 127.0.98.3:42069)
0929 18:31:03.129200 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:03.129212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:03.129333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:03.129599 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:03.129602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:03.129616 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:03.129738 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:03.129744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.130460 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.130465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:03.131246 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:03.131254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.131905 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.131909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:03.132001 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.132029 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:31:03.132116 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:31:03.132199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":314,"thread_start_us":140,"threads_started":1}
I20260929 18:31:03.132758 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7a7a15bdf4d0d8febc995f1f52095" candidate_uuid: "a15bd1a457204b3291a8c95b0c8c7039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" is_pre_election: true
I20260929 18:31:03.132889 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7a7a15bdf4d0d8febc995f1f52095" candidate_uuid: "a15bd1a457204b3291a8c95b0c8c7039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" is_pre_election: true
I20260929 18:31:03.133090 1107 raft_consensus.cc:2454] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15bd1a457204b3291a8c95b0c8c7039 in term 0.
I20260929 18:31:03.133249 1242 raft_consensus.cc:2454] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15bd1a457204b3291a8c95b0c8c7039 in term 0.
I20260929 18:31:03.133589 1310 leader_election.cc:302] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15bd1a457204b3291a8c95b0c8c7039, a8c83dac6e2e4d968afd32ffd2afd300; no voters:
I20260929 18:31:03.133860 1450 raft_consensus.cc:2781] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:03.133957 1450 raft_consensus.cc:482] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:03.134002 1450 raft_consensus.cc:3037] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:03.139417 1450 raft_consensus.cc:504] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:03.139837 1450 leader_election.cc:288] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [CANDIDATE]: Term 1 election: Requested vote from peers a8c83dac6e2e4d968afd32ffd2afd300 (127.0.98.1:46715), 030e2d6f3abc4ba892d57c0f9b7519db (127.0.98.2:46805)
I20260929 18:31:03.140187 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7a7a15bdf4d0d8febc995f1f52095" candidate_uuid: "a15bd1a457204b3291a8c95b0c8c7039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c83dac6e2e4d968afd32ffd2afd300"
I20260929 18:31:03.140251 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7a7a15bdf4d0d8febc995f1f52095" candidate_uuid: "a15bd1a457204b3291a8c95b0c8c7039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030e2d6f3abc4ba892d57c0f9b7519db"
I20260929 18:31:03.140316 1107 raft_consensus.cc:3037] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:03.140363 1242 raft_consensus.cc:3037] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:03.141614 1242 raft_consensus.cc:2454] T 2af7a7a15bdf4d0d8febc995f1f52095 P 030e2d6f3abc4ba892d57c0f9b7519db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15bd1a457204b3291a8c95b0c8c7039 in term 1.
I20260929 18:31:03.141614 1107 raft_consensus.cc:2454] T 2af7a7a15bdf4d0d8febc995f1f52095 P a8c83dac6e2e4d968afd32ffd2afd300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15bd1a457204b3291a8c95b0c8c7039 in term 1.
I20260929 18:31:03.141942 1311 leader_election.cc:302] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030e2d6f3abc4ba892d57c0f9b7519db, a15bd1a457204b3291a8c95b0c8c7039; no voters:
I20260929 18:31:03.142130 1450 raft_consensus.cc:2781] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:03.142359 1450 raft_consensus.cc:686] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [term 1 LEADER]: Becoming Leader. State: Replica: a15bd1a457204b3291a8c95b0c8c7039, State: Running, Role: LEADER
I20260929 18:31:03.142784 1450 consensus_queue.cc:237] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } }
I20260929 18:31:03.145947 961 catalog_manager.cc:5495] T 2af7a7a15bdf4d0d8febc995f1f52095 P a15bd1a457204b3291a8c95b0c8c7039 reported cstate change: term changed from 0 to 1, leader changed from <none> to a15bd1a457204b3291a8c95b0c8c7039 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a15bd1a457204b3291a8c95b0c8c7039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c83dac6e2e4d968afd32ffd2afd300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "030e2d6f3abc4ba892d57c0f9b7519db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15bd1a457204b3291a8c95b0c8c7039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45093 } health_report { overall_health: HEALTHY } } }
Sep 29 18:31:03 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:03.170464 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.159703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.0.98.1:46715)
0929 18:31:03.159852 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:03.159857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:03.160036 (+ 179us) server_negotiation.cc:407] Connection header received
0929 18:31:03.160207 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:03.160213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:03.160267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:03.160400 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:03.161454 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.161958 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.162781 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.163007 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.163117 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.165963 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:03.165974 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:03.165990 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:03.166014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:03.168531 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.169014 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.169020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.169022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.169103 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.169402 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.169407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.169410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.169674 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:03.169918 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:03.170128 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 18:31:03.170271 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 18:31:03 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:31:03 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:03.229696 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.220449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.0.98.1:46715)
0929 18:31:03.220583 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:03.220588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:03.220762 (+ 174us) server_negotiation.cc:407] Connection header received
0929 18:31:03.220975 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:03.220978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:03.221024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:03.221109 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:03.221807 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.222337 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.223172 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.223415 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.223524 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.226044 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:03.226047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:03.226049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:03.226072 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:03.227804 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.228257 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.228260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.228262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.228345 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.228612 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.228617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.228619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.228785 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:03.229062 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:03.229403 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:31:03.229531 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:31:03 dist-test-slave-fmv8 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:31:03 dist-test-slave-fmv8 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:03.314131 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:03.304803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.0.98.1:46715)
0929 18:31:03.304958 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:03.304963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:03.305160 (+ 197us) server_negotiation.cc:407] Connection header received
0929 18:31:03.305341 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:03.305344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:03.305402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:03.305492 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:03.306215 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.306760 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.307622 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:03.307844 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:03.307947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:03.310519 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:03.310522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:03.310525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:03.310549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:03.312515 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.312937 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.312942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.312944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.313024 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:03.313302 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:03.313306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:03.313315 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:03.313484 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:03.313620 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:03.313848 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:31:03.313964 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 18:31:03.315485 1067 tablet_service.cc:2908] Rejecting scan request for tablet 2af7a7a15bdf4d0d8febc995f1f52095: Uninitialized: clean time has not yet been initialized
I20260929 18:31:03.317788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1024
I20260929 18:31:03.324517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1159
I20260929 18:31:03.330252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1294
I20260929 18:31:03.335603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 930
2026-09-29T18:31:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5378 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:04 dist-test-slave-fmv8 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:04 dist-test-slave-fmv8 krb5kdc[1466](info): set up 2 sockets
Sep 29 18:31:04 dist-test-slave-fmv8 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 18:31:05.632066 392 mini_kdc.cc:121] Time spent starting KDC: real 2.273s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:05 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:05.749564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:05.858204 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:05.858469 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:05.858512 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:05.862669 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:05.862730 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:31:05.862747 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:05.862766 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:05.862811 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:05.862845 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:05.868062 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:05.869410 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:05.870761 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:05.879565 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:05.879565 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:05.879607 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:05.879611 1479 server_base.cc:1034] running on GCE node
I20260929 18:31:05.880400 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:05.881464 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:05.882654 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790706665882642 us; error 24 us; skew 500 ppm
Sep 29 18:31:05 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:05.889631 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:05.890549 1479 webserver.cc:460] Webserver started at http://127.0.98.62:33271/ using document root <none> and password file <none>
I20260929 18:31:05.891096 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:05.891223 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:05.891460 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:05.893517 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "d40c42f30fcc4368ac27e5a1466578d5"
format_stamp: "Formatted at 2026-09-29 18:31:05 on dist-test-slave-fmv8"
server_key: "986f828b8e5bb47705dccab5929e8a06"
server_key_iv: "2c90c26ada7affbcf0c0ce641e54ca05"
server_key_version: "encryptionkey@0"
I20260929 18:31:05.894068 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "d40c42f30fcc4368ac27e5a1466578d5"
format_stamp: "Formatted at 2026-09-29 18:31:05 on dist-test-slave-fmv8"
server_key: "986f828b8e5bb47705dccab5929e8a06"
server_key_iv: "2c90c26ada7affbcf0c0ce641e54ca05"
server_key_version: "encryptionkey@0"
I20260929 18:31:05.897634 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:05.899881 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:05.901051 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:05.901217 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "d40c42f30fcc4368ac27e5a1466578d5"
format_stamp: "Formatted at 2026-09-29 18:31:05 on dist-test-slave-fmv8"
server_key: "986f828b8e5bb47705dccab5929e8a06"
server_key_iv: "2c90c26ada7affbcf0c0ce641e54ca05"
server_key_version: "encryptionkey@0"
I20260929 18:31:05.901325 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:05.910754 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:05.913687 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:05.913892 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:05.921680 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42363
I20260929 18:31:05.921696 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42363 every 8 connection(s)
I20260929 18:31:05.922595 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:05.925472 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:05.925477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1479
I20260929 18:31:05.925587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:05.925853 392 external_mini_cluster.cc:1428] Setting key b245a8a1a4719e5d2ff6e09fb8b4a02c
I20260929 18:31:05.931640 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Bootstrap starting.
I20260929 18:31:05.934155 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Neither blocks nor log segments found. Creating new log.
Sep 29 18:31:05 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:05.934892 1557 log.cc:824] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:05.936815 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: No bootstrap required, opened a new log
I20260929 18:31:05.939815 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260929 18:31:05.940032 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:05.940068 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:05.927164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.62:42363)
0929 18:31:05.927580 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:05.927594 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:05.927627 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:31:05.928294 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:05.928312 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:05.928623 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:05.928950 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:05.929750 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:05.930711 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:05.931546 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:05.932058 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:05.932243 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:05.934897 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:05.934902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:05.934914 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:05.934939 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:05.937322 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:05.937788 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:05.937793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:05.937797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:05.937908 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:05.938204 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:05.938208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:05.938209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:05.938574 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:05.938771 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:05.939056 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:31:05.939333 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20260929 18:31:05.940110 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40c42f30fcc4368ac27e5a1466578d5, State: Initialized, Role: FOLLOWER
I20260929 18:31:05.940600 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260929 18:31:05.940766 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:05.940843 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:05.940945 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:05.941862 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260929 18:31:05.942252 1557 leader_election.cc:302] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40c42f30fcc4368ac27e5a1466578d5; no voters:
I20260929 18:31:05.942581 1557 leader_election.cc:288] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:05.942659 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:05.942885 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [term 1 LEADER]: Becoming Leader. State: Replica: d40c42f30fcc4368ac27e5a1466578d5, State: Running, Role: LEADER
I20260929 18:31:05.943218 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260929 18:31:05.943866 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:05.944990 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40c42f30fcc4368ac27e5a1466578d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } } }
I20260929 18:31:05.945114 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:05.945428 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40c42f30fcc4368ac27e5a1466578d5. Latest consensus state: current_term: 1 leader_uuid: "d40c42f30fcc4368ac27e5a1466578d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40c42f30fcc4368ac27e5a1466578d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } } }
I20260929 18:31:05.945503 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:05.945569 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:05.948751 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:05.954834 1571 catalog_manager.cc:1347] Generated new cluster ID: da737bedbab942c09dd7d119040417a8
I20260929 18:31:05.954919 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:05.970434 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:05.971531 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:05.979334 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Generated new TSK 0
I20260929 18:31:05.980100 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:06.015779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:06.126582 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:06.126830 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:06.126874 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:06.130887 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:06.130939 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:06.130977 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:06.131008 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:06.136008 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:06.136987 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:06.137910 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:06.147593 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:06.147621 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:06.147593 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:06.147934 1583 server_base.cc:1034] running on GCE node
I20260929 18:31:06.148530 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:06.149154 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:06.150344 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790706666150330 us; error 32 us; skew 500 ppm
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:06.153172 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:06.154008 1583 webserver.cc:460] Webserver started at http://127.0.98.1:41165/ using document root <none> and password file <none>
I20260929 18:31:06.154556 1583 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:06.154644 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:06.154865 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:06.156880 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "03f4d3e880534e98aef909ae74fa5a08"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "e5c1616389e52fcdc679b5c9b365936f"
server_key_iv: "6d7ea81d513933e2ac3240a0966ce051"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.157425 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03f4d3e880534e98aef909ae74fa5a08"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "e5c1616389e52fcdc679b5c9b365936f"
server_key_iv: "6d7ea81d513933e2ac3240a0966ce051"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.161084 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:06.163435 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.164819 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:06.164965 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "03f4d3e880534e98aef909ae74fa5a08"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "e5c1616389e52fcdc679b5c9b365936f"
server_key_iv: "6d7ea81d513933e2ac3240a0966ce051"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.165076 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:06.182776 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:06.185642 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:06.185787 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:06.187058 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:06.187144 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.187219 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:06.187264 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:06.197949 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34827
I20260929 18:31:06.198274 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34827 every 8 connection(s)
I20260929 18:31:06.198833 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:06.200852 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.187909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59573 (local address 127.0.98.62:42363)
0929 18:31:06.188055 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.188060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.188964 (+ 904us) server_negotiation.cc:407] Connection header received
0929 18:31:06.189936 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.189940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.190000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.190063 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.191729 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.192470 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.193583 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.193933 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.194056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.197011 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.197015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.197018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.197044 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.198752 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.199271 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.199274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.199276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.199357 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.199716 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.199719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.199721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.199898 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.200005 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.200564 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.200685 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:31:06.201582 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.188383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.1:59573)
0929 18:31:06.188805 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.188852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.189688 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.190559 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.190568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.190972 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.191552 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.191568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.192603 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.192606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.193449 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.193457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.194102 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.194106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.194306 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.194883 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.194898 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.196853 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.198874 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.198880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.198893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.199165 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.199447 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.199452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.199454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.199610 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.200020 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.200028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.200323 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.200502 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.200795 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
I20260929 18:31:06.201994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1583
I20260929 18:31:06.202132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:06.202454 392 external_mini_cluster.cc:1428] Setting key cfeb4b49a3cf05e7ec539fe3994fb945
I20260929 18:31:06.202926 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260929 18:31:06.203073 1714 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:06.203608 1714 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260929 18:31:06.206530 1510 ts_manager.cc:194] Registered new tserver with Master: 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827)
I20260929 18:31:06.207832 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59573
I20260929 18:31:06.213482 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.205059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.1:48053)
0929 18:31:06.205345 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.205362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.205454 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.206129 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.206134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.206383 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.206525 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.206535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.207539 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.207542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.208242 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.208254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.208888 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.208891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.208983 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.209533 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.209547 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.209962 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.211968 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.211971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.211972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.212245 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.212540 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.212542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.212544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.212621 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.213070 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.213072 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.213141 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.213237 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.213341 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260929 18:31:06.213619 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.205802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48053 (local address 127.0.98.62:42363)
0929 18:31:06.205909 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.205913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.205927 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:06.205959 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.205961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.206021 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.206097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.206704 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.207409 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.208358 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.208765 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.208886 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.210135 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.210141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.210144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.210170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.211848 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.212350 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.212353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.212355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.212443 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.212715 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.212718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.212720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.212970 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.213080 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.213323 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.213448 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:06.233124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:06.343879 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:06.344123 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:06.344238 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:06.348358 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:06.348417 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:06.348459 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:06.348498 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:06.353783 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:06.354738 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:06.355659 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:06.364890 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:06.365182 1723 server_base.cc:1034] running on GCE node
W20260929 18:31:06.364892 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:06.364989 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:06.365715 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:06.366343 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:06.367504 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790706666367489 us; error 32 us; skew 500 ppm
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:06.370579 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:06.371372 1723 webserver.cc:460] Webserver started at http://127.0.98.2:37555/ using document root <none> and password file <none>
I20260929 18:31:06.371886 1723 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:06.371942 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:06.372121 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:06.374181 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "367ab45e8bed4753b2a1add522247af5"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "5789b5da3d5bcfa587fa02b443d07e82"
server_key_iv: "60347ee609fc839d8a2ba84ce8e43bce"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.374691 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "367ab45e8bed4753b2a1add522247af5"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "5789b5da3d5bcfa587fa02b443d07e82"
server_key_iv: "60347ee609fc839d8a2ba84ce8e43bce"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.378427 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 18:31:06.380913 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.382215 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:06.382357 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "367ab45e8bed4753b2a1add522247af5"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "5789b5da3d5bcfa587fa02b443d07e82"
server_key_iv: "60347ee609fc839d8a2ba84ce8e43bce"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.382469 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:06.394243 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:06.397495 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:06.397699 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:06.399133 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:06.399204 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.399274 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:06.399317 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:06.410562 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36175
I20260929 18:31:06.410836 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36175 every 8 connection(s)
I20260929 18:31:06.411525 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:06.413645 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.399997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52477 (local address 127.0.98.62:42363)
0929 18:31:06.400139 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.400164 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.401120 (+ 956us) server_negotiation.cc:407] Connection header received
0929 18:31:06.402499 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.402506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.402587 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.402667 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.404794 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.405496 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.406732 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.406971 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.407095 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.409975 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.409978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.409980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.410036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.411611 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.412208 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.412212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.412214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.412296 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.412699 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.412702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.412703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.412875 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.412969 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.413421 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.413531 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:31:06.414410 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.400450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.2:52477)
0929 18:31:06.400937 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.400997 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.402213 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.403193 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.403202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.403720 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.404629 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.404652 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.405622 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.405626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.406600 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.406609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.407143 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.407147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.407328 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.407936 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.407952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.409805 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.411733 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.411739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.411753 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.412066 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.412398 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.412400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.412402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.412588 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.412979 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.412983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.413203 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.413345 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.413581 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260929 18:31:06.415652 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260929 18:31:06.415795 1854 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:06.416349 1854 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260929 18:31:06.417424 1511 ts_manager.cc:194] Registered new tserver with Master: 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175)
I20260929 18:31:06.418145 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52477
I20260929 18:31:06.419370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1723
I20260929 18:31:06.419481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:06.419778 392 external_mini_cluster.cc:1428] Setting key 7da39ff01771e58fadd0289e69fa54a8
I20260929 18:31:06.425695 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.417794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38083 (local address 127.0.98.62:42363)
0929 18:31:06.417899 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.417902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.418617 (+ 715us) server_negotiation.cc:407] Connection header received
0929 18:31:06.418730 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.418733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.418769 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.418835 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.419312 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.420081 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.421037 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.421345 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.421455 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.422552 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.422558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.422563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.422591 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.424049 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.424542 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.424545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.424546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.424613 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.424920 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.424923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.424924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.425143 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.425220 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.425507 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.425590 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:31:06.425905 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.417738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.2:38083)
0929 18:31:06.418532 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.418545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.418621 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.418875 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.418878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.419083 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.419188 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.419195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.420268 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.420272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.420885 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.420895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.421475 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.421477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.421564 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.422094 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.422105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.422405 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.424179 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.424185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.424190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.424449 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.424702 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.424708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.424713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.424814 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.425242 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.425248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.425346 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.425464 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.425624 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:06.447434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:06.559052 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:06.559317 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:06.559361 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:06.563750 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:06.563817 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:06.563856 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:06.563946 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:06.569309 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:06.570351 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:06.571358 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:06.580864 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:06.581023 1863 server_base.cc:1034] running on GCE node
W20260929 18:31:06.580906 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:06.580855 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:06.581525 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:06.582159 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:06.583346 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790706666583328 us; error 34 us; skew 500 ppm
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:06.586213 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:06.587149 1863 webserver.cc:460] Webserver started at http://127.0.98.3:34937/ using document root <none> and password file <none>
I20260929 18:31:06.587715 1863 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:06.587807 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:06.588047 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:06.590116 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ba17b18d8a74b4bb93bcf53c48383b2"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "d7dc771be89cef4d8bdda0212f4fc574"
server_key_iv: "64b4b05eabda4f453a4569e6761795d8"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.590693 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ba17b18d8a74b4bb93bcf53c48383b2"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "d7dc771be89cef4d8bdda0212f4fc574"
server_key_iv: "64b4b05eabda4f453a4569e6761795d8"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.594641 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:31:06.597144 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.598382 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:06.598560 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "3ba17b18d8a74b4bb93bcf53c48383b2"
format_stamp: "Formatted at 2026-09-29 18:31:06 on dist-test-slave-fmv8"
server_key: "d7dc771be89cef4d8bdda0212f4fc574"
server_key_iv: "64b4b05eabda4f453a4569e6761795d8"
server_key_version: "encryptionkey@0"
I20260929 18:31:06.598673 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:06.623521 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:06.626559 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:06.626755 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:06.628304 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:06.628372 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:06.628439 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:06.628485 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:06.639036 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41779
I20260929 18:31:06.639107 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41779 every 8 connection(s)
I20260929 18:31:06.639952 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:06.641959 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.629152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43477 (local address 127.0.98.62:42363)
0929 18:31:06.629299 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.629304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.630195 (+ 891us) server_negotiation.cc:407] Connection header received
0929 18:31:06.631295 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.631298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.631353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.631466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.633172 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.634081 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.635345 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.635583 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.635708 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.638176 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.638180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.638183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.638213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.639736 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.640360 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.640364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.640366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.640486 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.641010 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.641013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.641015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.641200 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.641281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.641740 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.641841 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:31:06.642762 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.629582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.3:43477)
0929 18:31:06.630030 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.630078 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.631055 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.631695 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.631705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.632231 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.632977 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.633000 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.634199 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.634204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.635219 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.635228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.635758 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.635762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.635933 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.636538 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.636555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.637997 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.639856 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.639863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.639877 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.640261 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.640729 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.640733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.640734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.640900 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.641294 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.641299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.641514 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.641658 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.641892 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260929 18:31:06.643851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 1863
I20260929 18:31:06.643985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:06.644039 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260929 18:31:06.644222 1994 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:06.644348 392 external_mini_cluster.cc:1428] Setting key fdf65d31c2b6c567a1f78a0b0565ef5e
I20260929 18:31:06.644901 1994 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260929 18:31:06.646031 1511 ts_manager.cc:194] Registered new tserver with Master: 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3:41779)
I20260929 18:31:06.646783 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43477
I20260929 18:31:06.653985 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.646391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.3:50005)
0929 18:31:06.646660 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.646681 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.646758 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.647074 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.647077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.647300 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:06.647410 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.647417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.648397 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.648399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.648975 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.648983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.649426 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.649429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.649506 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.649972 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:06.649983 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:06.650279 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:06.652113 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.652116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.652117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.652377 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.652857 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:06.652860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:06.652861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:06.652981 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:06.653358 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:06.653361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:06.653439 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.653714 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.653840 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260929 18:31:06.654028 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.646482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50005 (local address 127.0.98.62:42363)
0929 18:31:06.646609 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.646613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.646746 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:31:06.646837 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.646843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.646887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.646959 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.647527 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.648285 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.649080 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.649281 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.649364 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.650492 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.650497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.650501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.650526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.652002 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.652547 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.652550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.652551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.652629 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.653078 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.653081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.653082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.653249 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.653329 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.653792 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.653887 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:31:06.658100 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:31:06 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:06.683465 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.675183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.62:42363)
0929 18:31:06.675332 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.675336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.675351 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:06.675391 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.675393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.675442 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.675519 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:06.676302 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.676826 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.677694 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.677911 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.678058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.680527 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:06.680530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:06.680532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:06.680557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:06.681797 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.682238 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.682243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.682244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.682323 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:06.682591 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:06.682596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:06.682598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:06.682776 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:06.682934 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.683169 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.683293 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:31:06.686625 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:06.689697 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:06.704134 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.698773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34827 (local address 127.0.0.1:51496)
0929 18:31:06.699025 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.699039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.699138 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.699868 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.699871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.699888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:06.699983 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.699989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.701289 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.701293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.702486 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.702495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.703545 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.703550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.703659 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.703724 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.703849 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.703975 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260929 18:31:06.704129 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.699009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.98.1:34827)
0929 18:31:06.699373 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.699389 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.699411 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:31:06.699503 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.699509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.699712 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.700118 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:06.700303 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.701176 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.702627 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.703436 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.703542 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.703705 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.703882 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.703984 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
I20260929 18:31:06.704134 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.697999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36175 (local address 127.0.0.1:34586)
0929 18:31:06.698331 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.698377 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.698670 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.699521 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.699527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.699552 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:06.699691 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.699702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.701151 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.701155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.702490 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.702499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.703732 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.703735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.703860 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.703882 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.703934 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.704002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":30,"thread_start_us":108,"threads_started":1}
I20260929 18:31:06.704919 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.698141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.2:36175)
0929 18:31:06.698458 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.698469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.698494 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:31:06.698891 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.698898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.699180 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.699341 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:06.699831 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.701037 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.702650 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.703543 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.703641 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.704368 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.704418 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.704508 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":124,"threads_started":1}
I20260929 18:31:06.705036 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.699586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.3:41779)
0929 18:31:06.700284 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.700295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.700315 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:31:06.700381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.700386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.700548 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.700694 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:06.700916 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.702245 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.703678 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.704507 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.704633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.704779 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.704831 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.704911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":103,"threads_started":1}
I20260929 18:31:06.705073 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.699063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41779 (local address 127.0.0.1:50040)
0929 18:31:06.699941 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.699954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.700034 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.700698 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.700701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.700718 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:06.700803 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.700809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.702365 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.702368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.703541 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.703550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.704626 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.704632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.704732 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.704754 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.704814 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.704879 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":84,"threads_started":1}
I20260929 18:31:06.707348 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 (DEFAULT_TABLE table=test-table [id=8dea31b5a55041c985e23640ba41df84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:06.707654 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 (DEFAULT_TABLE table=test-table [id=8dea31b5a55041c985e23640ba41df84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:06.708365 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1294db993245cf910e45ff8beb7ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:06.708137 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 (DEFAULT_TABLE table=test-table [id=8dea31b5a55041c985e23640ba41df84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:06.708671 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1294db993245cf910e45ff8beb7ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:06.709097 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1294db993245cf910e45ff8beb7ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:06.714752 2017 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Bootstrap starting.
I20260929 18:31:06.716113 2018 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Bootstrap starting.
I20260929 18:31:06.716846 2019 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Bootstrap starting.
I20260929 18:31:06.717082 2017 tablet_bootstrap.cc:654] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:06.718400 2017 log.cc:824] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:06.718511 2019 tablet_bootstrap.cc:654] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:06.718559 2018 tablet_bootstrap.cc:654] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:06.719319 2019 log.cc:824] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:06.719349 2018 log.cc:824] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:06.720325 2017 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: No bootstrap required, opened a new log
I20260929 18:31:06.720533 2017 ts_tablet_manager.cc:1397] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:31:06.721027 2019 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: No bootstrap required, opened a new log
I20260929 18:31:06.721156 2019 ts_tablet_manager.cc:1397] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 18:31:06.721704 2018 tablet_bootstrap.cc:492] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: No bootstrap required, opened a new log
I20260929 18:31:06.721858 2018 ts_tablet_manager.cc:1397] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 18:31:06.723737 2019 raft_consensus.cc:348] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.723923 2019 raft_consensus.cc:374] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:06.723953 2019 raft_consensus.cc:729] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Initialized, Role: FOLLOWER
I20260929 18:31:06.724444 2019 consensus_queue.cc:260] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.724613 2017 raft_consensus.cc:348] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.725039 2017 raft_consensus.cc:374] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:06.725137 2017 raft_consensus.cc:729] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f4d3e880534e98aef909ae74fa5a08, State: Initialized, Role: FOLLOWER
I20260929 18:31:06.725268 1994 heartbeater.cc:503] Master 127.0.98.62:42363 was elected leader, sending a full tablet report...
I20260929 18:31:06.725608 2017 consensus_queue.cc:260] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.725608 2018 raft_consensus.cc:348] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.725909 2018 raft_consensus.cc:374] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:06.726011 2018 raft_consensus.cc:729] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367ab45e8bed4753b2a1add522247af5, State: Initialized, Role: FOLLOWER
I20260929 18:31:06.726334 2019 ts_tablet_manager.cc:1428] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:06.726531 2017 ts_tablet_manager.cc:1428] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:31:06.726506 2018 consensus_queue.cc:260] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.726909 1714 heartbeater.cc:503] Master 127.0.98.62:42363 was elected leader, sending a full tablet report...
I20260929 18:31:06.727226 1854 heartbeater.cc:503] Master 127.0.98.62:42363 was elected leader, sending a full tablet report...
I20260929 18:31:06.727386 2018 ts_tablet_manager.cc:1428] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:31:06.891032 1995 tablet.cc:2369] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:06.912801 1855 tablet.cc:2369] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:06.950161 1715 tablet.cc:2369] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:06.959602 2023 raft_consensus.cc:482] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:06.959975 2023 raft_consensus.cc:504] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.961244 2023 leader_election.cc:288] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175), 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827)
I20260929 18:31:06.965627 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.961615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36175 (local address 127.0.98.3:40407)
0929 18:31:06.961771 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.961787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.961901 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.962355 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.962360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.962386 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:06.962535 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.962543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.963395 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.963401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.964592 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.964603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.965270 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.965273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.965355 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.965375 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.965440 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.965513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 18:31:06.965688 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.961756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40407 (local address 127.0.98.2:36175)
0929 18:31:06.961973 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.961979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.961996 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:31:06.962053 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.962056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.962140 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.962238 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:06.962689 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.963271 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.964722 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.965111 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.965243 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.965313 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.965476 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.965554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:31:06.966377 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.962214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60013 (local address 127.0.98.1:34827)
0929 18:31:06.962354 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:06.962359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:06.962500 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:31:06.962556 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:06.962559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:06.962651 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:06.962748 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:06.963401 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.963912 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.965214 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.965769 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.965874 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.965941 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:06.966214 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 18:31:06.966282 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:31:06.966380 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:06.961980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34827 (local address 127.0.98.3:60013)
0929 18:31:06.962347 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:06.962361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:06.962476 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:06.962768 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:06.962770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:06.962786 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:06.963269 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.963276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.964030 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.964033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.965093 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:06.965101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:06.965952 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:06.965959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:06.966051 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:06.966070 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:31:06.966176 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:31:06.966242 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":1,"thread_start_us":181,"threads_started":1}
I20260929 18:31:06.966435 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1294db993245cf910e45ff8beb7ff9" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5" is_pre_election: true
I20260929 18:31:06.966820 1808 raft_consensus.cc:2454] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:06.967288 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1294db993245cf910e45ff8beb7ff9" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08" is_pre_election: true
I20260929 18:31:06.967360 1879 leader_election.cc:302] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367ab45e8bed4753b2a1add522247af5, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:06.967599 1668 raft_consensus.cc:2454] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:06.967655 2023 raft_consensus.cc:2781] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:06.967747 2023 raft_consensus.cc:482] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:06.967798 2023 raft_consensus.cc:3037] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:06.968969 2023 raft_consensus.cc:504] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.969369 2023 leader_election.cc:288] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Requested vote from peers 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175), 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827)
I20260929 18:31:06.969779 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1294db993245cf910e45ff8beb7ff9" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5"
I20260929 18:31:06.969775 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1294db993245cf910e45ff8beb7ff9" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08"
I20260929 18:31:06.969911 1808 raft_consensus.cc:3037] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:06.969954 1668 raft_consensus.cc:3037] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:06.970898 1808 raft_consensus.cc:2454] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:06.970988 1668 raft_consensus.cc:2454] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:06.971213 1879 leader_election.cc:302] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367ab45e8bed4753b2a1add522247af5, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:06.971385 2023 raft_consensus.cc:2781] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:06.971611 2023 raft_consensus.cc:686] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Running, Role: LEADER
I20260929 18:31:06.971951 2023 consensus_queue.cc:237] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:06.975301 1511 catalog_manager.cc:5495] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } health_report { overall_health: HEALTHY } } }
I20260929 18:31:07.006546 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.001994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.0.98.3:41779)
0929 18:31:07.002124 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.002127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.002142 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:07.002477 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.002480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.002553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.002646 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:07.003105 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.003722 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.004862 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.005106 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.005264 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.005692 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:07.006118 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:07.006210 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.006347 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.006433 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:31:07.012629 1668 raft_consensus.cc:1261] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.012629 1808 raft_consensus.cc:1261] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.013490 2027 consensus_queue.cc:1035] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.013809 2023 consensus_queue.cc:1035] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.022024 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334734508076646400 to 7334734507920269312. Current Snapshot: MvccSnapshot[applied={T|T < 7334734508076646400}]
I20260929 18:31:07.022043 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334734508076646400 to 7334734507920269312. Current Snapshot: MvccSnapshot[applied={T|T < 7334734508076646400}]
I20260929 18:31:07.033670 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:07.033883 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:07.036641 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Sending DeleteTablet for 3 replicas of tablet 0b1294db993245cf910e45ff8beb7ff9
I20260929 18:31:07.037530 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34586
I20260929 18:31:07.037647 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260929 18:31:07.037695 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1294db993245cf910e45ff8beb7ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50040
I20260929 18:31:07.038192 2040 tablet_replica.cc:330] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: stopping tablet replica
I20260929 18:31:07.038230 2041 tablet_replica.cc:330] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: stopping tablet replica
I20260929 18:31:07.038575 2041 raft_consensus.cc:2229] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:07.038585 2040 raft_consensus.cc:2229] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:07.038975 2040 raft_consensus.cc:2258] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:07.038937 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:31:07.039127 2041 raft_consensus.cc:2258] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:31:07.039362 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:07.039577 2042 tablet_replica.cc:330] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: stopping tablet replica
I20260929 18:31:07.039860 2042 raft_consensus.cc:2229] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:07.040171 2042 raft_consensus.cc:2258] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:07.040724 2040 ts_tablet_manager.cc:1905] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.041694 2042 ts_tablet_manager.cc:1905] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.041916 2041 ts_tablet_manager.cc:1905] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.044904 2042 ts_tablet_manager.cc:1918] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:07.045110 2042 log.cc:1196] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/0b1294db993245cf910e45ff8beb7ff9
I20260929 18:31:07.045593 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 966240af691a489ab4281232c3994868 (DEFAULT_TABLE table=test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:07.045866 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966240af691a489ab4281232c3994868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.045794 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 966240af691a489ab4281232c3994868 (DEFAULT_TABLE table=test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:07.046406 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 966240af691a489ab4281232c3994868 (DEFAULT_TABLE table=test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:07.046581 2040 ts_tablet_manager.cc:1918] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:07.046638 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966240af691a489ab4281232c3994868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.046660 2040 log.cc:1196] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/0b1294db993245cf910e45ff8beb7ff9
I20260929 18:31:07.046986 2040 ts_tablet_manager.cc:1939] T 0b1294db993245cf910e45ff8beb7ff9 P 367ab45e8bed4753b2a1add522247af5: Deleting consensus metadata
I20260929 18:31:07.048277 2017 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Bootstrap starting.
I20260929 18:31:07.045575 2042 ts_tablet_manager.cc:1939] T 0b1294db993245cf910e45ff8beb7ff9 P 03f4d3e880534e98aef909ae74fa5a08: Deleting consensus metadata
I20260929 18:31:07.049240 2017 tablet_bootstrap.cc:654] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.049336 2041 ts_tablet_manager.cc:1918] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:07.049428 2041 log.cc:1196] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/0b1294db993245cf910e45ff8beb7ff9
I20260929 18:31:07.049784 2041 ts_tablet_manager.cc:1939] T 0b1294db993245cf910e45ff8beb7ff9 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting consensus metadata
I20260929 18:31:07.049921 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966240af691a489ab4281232c3994868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.050621 2017 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: No bootstrap required, opened a new log
I20260929 18:31:07.050719 2017 ts_tablet_manager.cc:1397] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:07.051170 2017 raft_consensus.cc:348] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.051337 2017 raft_consensus.cc:374] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.051385 2017 raft_consensus.cc:729] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f4d3e880534e98aef909ae74fa5a08, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.051425 1495 catalog_manager.cc:4841] TS 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3:41779): tablet 0b1294db993245cf910e45ff8beb7ff9 (table test-table [id=8dea31b5a55041c985e23640ba41df84]) successfully deleted
I20260929 18:31:07.051505 2017 consensus_queue.cc:260] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.051702 1495 catalog_manager.cc:4841] TS 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827): tablet 0b1294db993245cf910e45ff8beb7ff9 (table test-table [id=8dea31b5a55041c985e23640ba41df84]) successfully deleted
I20260929 18:31:07.051776 2017 ts_tablet_manager.cc:1428] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:07.051816 1495 catalog_manager.cc:4841] TS 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175): tablet 0b1294db993245cf910e45ff8beb7ff9 (table test-table [id=8dea31b5a55041c985e23640ba41df84]) successfully deleted
I20260929 18:31:07.052316 2019 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Bootstrap starting.
I20260929 18:31:07.052783 2018 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Bootstrap starting.
I20260929 18:31:07.053318 2019 tablet_bootstrap.cc:654] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.053786 2018 tablet_bootstrap.cc:654] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.055315 2019 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: No bootstrap required, opened a new log
I20260929 18:31:07.055447 2019 ts_tablet_manager.cc:1397] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:07.055861 2018 tablet_bootstrap.cc:492] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: No bootstrap required, opened a new log
I20260929 18:31:07.055897 2019 raft_consensus.cc:348] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.055974 2018 ts_tablet_manager.cc:1397] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:31:07.056015 2019 raft_consensus.cc:374] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.056058 2019 raft_consensus.cc:729] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.056293 2019 consensus_queue.cc:260] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.056476 2018 raft_consensus.cc:348] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.056571 2018 raft_consensus.cc:374] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.056593 2019 ts_tablet_manager.cc:1428] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:07.056604 2018 raft_consensus.cc:729] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367ab45e8bed4753b2a1add522247af5, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.056751 2018 consensus_queue.cc:260] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.057009 2018 ts_tablet_manager.cc:1428] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:07.241777 2035 raft_consensus.cc:482] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:07.241925 2035 raft_consensus.cc:504] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.242388 2035 leader_election.cc:288] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827), 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175)
I20260929 18:31:07.243009 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966240af691a489ab4281232c3994868" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08" is_pre_election: true
I20260929 18:31:07.243053 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966240af691a489ab4281232c3994868" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5" is_pre_election: true
I20260929 18:31:07.243156 1668 raft_consensus.cc:2454] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:07.243264 1808 raft_consensus.cc:2454] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:07.243585 1879 leader_election.cc:302] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f4d3e880534e98aef909ae74fa5a08, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:07.243789 2035 raft_consensus.cc:2781] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:07.243891 2035 raft_consensus.cc:482] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:07.243964 2035 raft_consensus.cc:3037] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.245417 2035 raft_consensus.cc:504] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.245857 2035 leader_election.cc:288] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Requested vote from peers 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827), 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175)
I20260929 18:31:07.246358 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966240af691a489ab4281232c3994868" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08"
I20260929 18:31:07.246521 1668 raft_consensus.cc:3037] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.246480 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966240af691a489ab4281232c3994868" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5"
I20260929 18:31:07.246613 1808 raft_consensus.cc:3037] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.247761 1668 raft_consensus.cc:2454] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:07.247776 1808 raft_consensus.cc:2454] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:07.248137 1879 leader_election.cc:302] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f4d3e880534e98aef909ae74fa5a08, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:07.248437 2035 raft_consensus.cc:2781] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:07.248528 2035 raft_consensus.cc:686] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Running, Role: LEADER
I20260929 18:31:07.248677 2035 consensus_queue.cc:237] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.250411 1509 catalog_manager.cc:5495] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } health_report { overall_health: HEALTHY } } }
Sep 29 18:31:07 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:07.335099 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.325959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.0.1:40828)
0929 18:31:07.326436 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:07.326455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:07.326606 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:07.326898 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:07.326902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:07.327353 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:07.327450 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.327457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.328395 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.328400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.329004 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.329011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.329495 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.329498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.329577 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.330156 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:07.330168 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:07.331697 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:07.333431 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:07.333442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:07.333449 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:07.333686 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:07.333992 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:07.333997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:07.334000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:07.334084 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:07.334482 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:07.334490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:07.334606 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 18:31:07.334725 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:31:07.334887 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":122,"threads_started":1}
I20260929 18:31:07.335098 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.326083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.98.62:42363)
0929 18:31:07.326412 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.326422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.326548 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:31:07.326722 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.326726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.326774 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.326887 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:07.327579 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.328271 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.329127 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.329376 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.329494 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.331846 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:07.331850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:07.331852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:07.331906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:07.333307 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.333791 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.333798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.333800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.333891 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.334187 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.334191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.334193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.334370 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:07.334483 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.334784 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.334955 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":269,"thread_start_us":76,"threads_started":1}
I20260929 18:31:07.338922 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:31:07.339682 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:07.344630 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 22e8872017b8422285553498b0ba11e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19fccfd933147d2ab00ebc7255b3f9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:07.344645 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 22e8872017b8422285553498b0ba11e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19fccfd933147d2ab00ebc7255b3f9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:07.344731 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 22e8872017b8422285553498b0ba11e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e19fccfd933147d2ab00ebc7255b3f9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:07.345043 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e8872017b8422285553498b0ba11e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.345043 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e8872017b8422285553498b0ba11e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.345171 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e8872017b8422285553498b0ba11e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:07.347790 2019 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: Bootstrap starting.
I20260929 18:31:07.347925 2017 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: Bootstrap starting.
I20260929 18:31:07.348657 2019 tablet_bootstrap.cc:654] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.348802 2017 tablet_bootstrap.cc:654] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.349730 2018 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: Bootstrap starting.
I20260929 18:31:07.349881 2019 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: No bootstrap required, opened a new log
I20260929 18:31:07.349990 2019 ts_tablet_manager.cc:1397] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:07.350399 2019 raft_consensus.cc:348] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.350519 2019 raft_consensus.cc:374] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.350565 2019 raft_consensus.cc:729] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.350687 2018 tablet_bootstrap.cc:654] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:07.350692 2019 consensus_queue.cc:260] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.350977 2019 ts_tablet_manager.cc:1428] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:31:07.350919 2035 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.351054 2035 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.351583 2017 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: No bootstrap required, opened a new log
I20260929 18:31:07.351667 2017 ts_tablet_manager.cc:1397] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:31:07.352133 2017 raft_consensus.cc:348] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.352293 2017 raft_consensus.cc:374] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.352335 2017 raft_consensus.cc:729] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f4d3e880534e98aef909ae74fa5a08, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.352449 2017 consensus_queue.cc:260] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.352723 2017 ts_tablet_manager.cc:1428] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:31:07.353006 2024 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.353143 2024 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.353641 2018 tablet_bootstrap.cc:492] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: No bootstrap required, opened a new log
I20260929 18:31:07.353729 2018 ts_tablet_manager.cc:1397] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:31:07.354141 2018 raft_consensus.cc:348] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.354257 2018 raft_consensus.cc:374] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:07.354293 2018 raft_consensus.cc:729] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367ab45e8bed4753b2a1add522247af5, State: Initialized, Role: FOLLOWER
I20260929 18:31:07.354409 2018 consensus_queue.cc:260] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.354707 2018 ts_tablet_manager.cc:1428] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:07.354691 2025 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.354851 2025 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.393029 2035 raft_consensus.cc:482] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:07.393158 2035 raft_consensus.cc:504] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.393646 2035 leader_election.cc:288] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827), 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175)
I20260929 18:31:07.394061 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e8872017b8422285553498b0ba11e6" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08" is_pre_election: true
I20260929 18:31:07.394133 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e8872017b8422285553498b0ba11e6" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5" is_pre_election: true
I20260929 18:31:07.394222 1668 raft_consensus.cc:2454] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:07.394327 1808 raft_consensus.cc:2454] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 0.
I20260929 18:31:07.394536 1879 leader_election.cc:302] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f4d3e880534e98aef909ae74fa5a08, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:07.394711 2035 raft_consensus.cc:2781] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:07.394766 2035 raft_consensus.cc:482] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:07.394821 2035 raft_consensus.cc:3037] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.395917 2035 raft_consensus.cc:504] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.396312 2035 leader_election.cc:288] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Requested vote from peers 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827), 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175)
I20260929 18:31:07.396708 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e8872017b8422285553498b0ba11e6" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ab45e8bed4753b2a1add522247af5"
I20260929 18:31:07.396703 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e8872017b8422285553498b0ba11e6" candidate_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4d3e880534e98aef909ae74fa5a08"
I20260929 18:31:07.396796 1808 raft_consensus.cc:3037] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.396806 1668 raft_consensus.cc:3037] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:07.397568 1808 raft_consensus.cc:2454] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:07.397732 1668 raft_consensus.cc:2454] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba17b18d8a74b4bb93bcf53c48383b2 in term 1.
I20260929 18:31:07.397838 1879 leader_election.cc:302] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367ab45e8bed4753b2a1add522247af5, 3ba17b18d8a74b4bb93bcf53c48383b2; no voters:
I20260929 18:31:07.398007 2035 raft_consensus.cc:2781] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:07.398099 2035 raft_consensus.cc:686] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba17b18d8a74b4bb93bcf53c48383b2, State: Running, Role: LEADER
I20260929 18:31:07.398250 2035 consensus_queue.cc:237] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } }
I20260929 18:31:07.398955 2023 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: TxnStatusTablet state changed. Reason: New leader 3ba17b18d8a74b4bb93bcf53c48383b2. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.399067 2023 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:07.399292 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:31:07.399492 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:31:07.399997 1510 catalog_manager.cc:5495] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } health_report { overall_health: HEALTHY } } }
I20260929 18:31:07.415447 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.411477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.0.98.3:41779)
0929 18:31:07.411605 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.411610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.411624 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:07.411670 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.411673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.411720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.411803 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:07.412133 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.412905 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.414222 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.414545 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.414649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.414971 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:07.415050 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:07.415127 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.415260 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.415318 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:31:07.415474 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.411249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41779 (local address 127.0.0.1:50062)
0929 18:31:07.411386 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:07.411404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:07.411515 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:07.411842 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:07.411845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:07.411856 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:07.411993 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.412012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.413023 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.413027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.414085 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.414095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.414660 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.414664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.414759 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.414812 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:07.415146 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:07.415160 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:31:07.415234 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:31:07.415322 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:31:07.417985 1668 raft_consensus.cc:1261] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.418046 1808 raft_consensus.cc:1261] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.418468 2035 consensus_queue.cc:1035] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.418623 2023 consensus_queue.cc:1035] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.420758 2025 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: TxnStatusTablet state changed. Reason: New leader 3ba17b18d8a74b4bb93bcf53c48383b2. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.420888 2025 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.421386 2024 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: TxnStatusTablet state changed. Reason: New leader 3ba17b18d8a74b4bb93bcf53c48383b2. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.421514 2024 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.428769 2027 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.428892 2027 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:07.429193 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334734509739937792 to 7334734509664722944. Current Snapshot: MvccSnapshot[applied={T|T < 7334734509739937792 or (T in {7334734509739937792})}]
I20260929 18:31:07.429636 2024 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.429756 2024 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 03f4d3e880534e98aef909ae74fa5a08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.431155 2035 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.431269 2035 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 3ba17b18d8a74b4bb93bcf53c48383b2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:07.432008 2025 tablet_replica.cc:439] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 } } peers { permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 } } peers { permanent_uuid: "3ba17b18d8a74b4bb93bcf53c48383b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41779 } } }
I20260929 18:31:07.432130 2025 tablet_replica.cc:442] T 22e8872017b8422285553498b0ba11e6 P 367ab45e8bed4753b2a1add522247af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:07.444505 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.440882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38457 (local address 127.0.98.3:41779)
0929 18:31:07.441005 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.441009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.441210 (+ 201us) server_negotiation.cc:407] Connection header received
0929 18:31:07.441316 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.441319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.441367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.441446 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:07.441645 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.442410 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.443251 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.443601 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.443703 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.443963 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:07.444044 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:07.444118 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.444240 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.444317 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":45}
I20260929 18:31:07.444576 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.440798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41779 (local address 127.0.98.3:38457)
0929 18:31:07.441132 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:07.441145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:07.441232 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:07.441464 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:07.441466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:07.441471 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:07.441553 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.441557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.442519 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.442521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.443144 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:07.443152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.443705 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.443708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:07.443768 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.443830 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:07.444139 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:07.444145 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:31:07.444230 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:31:07.444323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":13,"thread_start_us":126,"threads_started":1}
I20260929 18:31:07.452056 1808 raft_consensus.cc:1261] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.452283 1668 raft_consensus.cc:1261] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Refusing update from remote peer 3ba17b18d8a74b4bb93bcf53c48383b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:07.452463 2066 consensus_queue.cc:1035] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367ab45e8bed4753b2a1add522247af5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.452672 2027 consensus_queue.cc:1035] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f4d3e880534e98aef909ae74fa5a08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:07.490032 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:07.490214 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:07.492301 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d40c42f30fcc4368ac27e5a1466578d5: Sending DeleteTablet for 3 replicas of tablet 966240af691a489ab4281232c3994868
I20260929 18:31:07.493010 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 966240af691a489ab4281232c3994868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260929 18:31:07.493131 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 966240af691a489ab4281232c3994868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34586
I20260929 18:31:07.493225 2042 tablet_replica.cc:330] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: stopping tablet replica
I20260929 18:31:07.493207 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 966240af691a489ab4281232c3994868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50040
I20260929 18:31:07.493371 2042 raft_consensus.cc:2229] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:07.493417 2041 tablet_replica.cc:330] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: stopping tablet replica
I20260929 18:31:07.493495 2042 raft_consensus.cc:2258] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:07.493527 2041 raft_consensus.cc:2229] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:07.493666 2040 tablet_replica.cc:330] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: stopping tablet replica
I20260929 18:31:07.493714 2041 raft_consensus.cc:2258] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:07.493782 2040 raft_consensus.cc:2229] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:07.493911 2040 raft_consensus.cc:2258] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:07.494870 2042 ts_tablet_manager.cc:1905] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.494884 2041 ts_tablet_manager.cc:1905] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.497464 2042 ts_tablet_manager.cc:1918] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:07.497566 2042 log.cc:1196] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/966240af691a489ab4281232c3994868
I20260929 18:31:07.497579 2041 ts_tablet_manager.cc:1918] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:07.497643 2041 log.cc:1196] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/966240af691a489ab4281232c3994868
I20260929 18:31:07.497852 2042 ts_tablet_manager.cc:1939] T 966240af691a489ab4281232c3994868 P 03f4d3e880534e98aef909ae74fa5a08: Deleting consensus metadata
I20260929 18:31:07.497906 2041 ts_tablet_manager.cc:1939] T 966240af691a489ab4281232c3994868 P 3ba17b18d8a74b4bb93bcf53c48383b2: Deleting consensus metadata
I20260929 18:31:07.498570 1495 catalog_manager.cc:4841] TS 03f4d3e880534e98aef909ae74fa5a08 (127.0.98.1:34827): tablet 966240af691a489ab4281232c3994868 (table test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]) successfully deleted
I20260929 18:31:07.498764 1495 catalog_manager.cc:4841] TS 3ba17b18d8a74b4bb93bcf53c48383b2 (127.0.98.3:41779): tablet 966240af691a489ab4281232c3994868 (table test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]) successfully deleted
I20260929 18:31:07.498989 2040 ts_tablet_manager.cc:1905] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:07.501540 2040 ts_tablet_manager.cc:1918] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:07.501619 2040 log.cc:1196] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/966240af691a489ab4281232c3994868
I20260929 18:31:07.501850 2040 ts_tablet_manager.cc:1939] T 966240af691a489ab4281232c3994868 P 367ab45e8bed4753b2a1add522247af5: Deleting consensus metadata
I20260929 18:31:07.502504 1495 catalog_manager.cc:4841] TS 367ab45e8bed4753b2a1add522247af5 (127.0.98.2:36175): tablet 966240af691a489ab4281232c3994868 (table test-table [id=b2bceb50d5414ec6b35a0d4f7a2b8bde]) successfully deleted
Sep 29 18:31:07 dist-test-slave-fmv8 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706666, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:07.512895 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.502462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.0.98.1:34827)
0929 18:31:07.502674 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.502679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.502693 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:07.502730 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.502733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.502776 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.502886 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:07.503822 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.504644 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.505499 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.505848 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.505960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.509177 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:07.509189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:07.509206 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:07.509231 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:07.511126 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.511531 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.511538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.511540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.511633 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.511919 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.511922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.511924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.512241 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:07.512458 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.512574 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.512697 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
W20260929 18:31:07.513383 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51502
I20260929 18:31:07.523028 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:07.515050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.98.62:42363)
0929 18:31:07.515184 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:07.515188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:07.515379 (+ 191us) server_negotiation.cc:407] Connection header received
0929 18:31:07.515454 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:07.515457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:07.515503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:07.515587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:07.516268 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.517050 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.517797 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:07.518157 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:07.518259 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:07.519530 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:07.519534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:07.519536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:07.519562 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:07.521502 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.521931 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.521934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.521936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.522020 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:07.522260 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:07.522263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:07.522265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:07.522460 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:07.522585 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:07.522747 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:31:07.522871 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:31:07.523396 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40832
I20260929 18:31:07.524569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1583
I20260929 18:31:07.530817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1723
I20260929 18:31:07.536763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1863
I20260929 18:31:07.543155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 1479
2026-09-29T18:31:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4206 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:07 dist-test-slave-fmv8 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:07 dist-test-slave-fmv8 krb5kdc[2077](info): set up 2 sockets
Sep 29 18:31:07 dist-test-slave-fmv8 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260929 18:31:08.588006 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:08 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:08.726791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:08.838806 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:08.839057 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:08.839097 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:08.843350 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:08.843415 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:31:08.843433 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:08.843452 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:08.843490 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:08.843525 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:08.848839 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43679
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:08.849951 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:08.850965 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:08.859319 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:08.859336 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:08.859382 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:08.859644 2090 server_base.cc:1034] running on GCE node
I20260929 18:31:08.860191 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:08.861292 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:08.862458 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790706668862440 us; error 29 us; skew 500 ppm
Sep 29 18:31:08 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:08.865353 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:08.866283 2090 webserver.cc:460] Webserver started at http://127.0.98.62:42339/ using document root <none> and password file <none>
I20260929 18:31:08.866830 2090 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:08.866922 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:08.867153 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:08.869246 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "6218658cc659415096f8b1a880ae7f03"
format_stamp: "Formatted at 2026-09-29 18:31:08 on dist-test-slave-fmv8"
server_key: "f8990fbb76d87987d1693657b61c9c1f"
server_key_iv: "7c2ab24cefef39113638581a6a8b761f"
server_key_version: "encryptionkey@0"
I20260929 18:31:08.869807 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "6218658cc659415096f8b1a880ae7f03"
format_stamp: "Formatted at 2026-09-29 18:31:08 on dist-test-slave-fmv8"
server_key: "f8990fbb76d87987d1693657b61c9c1f"
server_key_iv: "7c2ab24cefef39113638581a6a8b761f"
server_key_version: "encryptionkey@0"
I20260929 18:31:08.873463 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:08.875926 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:08.877207 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:08.877359 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "6218658cc659415096f8b1a880ae7f03"
format_stamp: "Formatted at 2026-09-29 18:31:08 on dist-test-slave-fmv8"
server_key: "f8990fbb76d87987d1693657b61c9c1f"
server_key_iv: "7c2ab24cefef39113638581a6a8b761f"
server_key_version: "encryptionkey@0"
I20260929 18:31:08.877478 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:08.887203 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:08.890336 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:08.890529 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:08.898178 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43679
I20260929 18:31:08.898216 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43679 every 8 connection(s)
I20260929 18:31:08.899061 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:08.901666 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:08.903033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2090
I20260929 18:31:08.903128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:08.903419 392 external_mini_cluster.cc:1428] Setting key d2b325915cf253adfb431c7d9c36b635
I20260929 18:31:08.907389 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Bootstrap starting.
I20260929 18:31:08.909569 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:08.910331 2168 log.cc:824] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:31:08 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:08.912402 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: No bootstrap required, opened a new log
I20260929 18:31:08.915505 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } }
I20260929 18:31:08.915715 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:08.915807 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6218658cc659415096f8b1a880ae7f03, State: Initialized, Role: FOLLOWER
I20260929 18:31:08.916272 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } }
I20260929 18:31:08.916421 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:08.916483 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:08.916590 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:08.917528 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } }
I20260929 18:31:08.917834 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:08.904791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.0.98.62:43679)
0929 18:31:08.905198 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:08.905213 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:08.905247 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:31:08.905862 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:08.905878 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:08.906201 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:08.906536 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:08.907321 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:08.908394 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:08.909258 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:08.909770 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:08.909932 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:08.912669 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:08.912674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:08.912688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:08.912712 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:08.914947 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:08.915387 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:08.915398 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:08.915404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:08.915510 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:08.915822 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:08.915826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:08.915827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:08.916352 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:08.916562 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:08.916791 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:31:08.917077 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260929 18:31:08.917874 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6218658cc659415096f8b1a880ae7f03; no voters:
I20260929 18:31:08.918218 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:08.918286 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:08.918467 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [term 1 LEADER]: Becoming Leader. State: Replica: 6218658cc659415096f8b1a880ae7f03, State: Running, Role: LEADER
I20260929 18:31:08.918741 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } }
I20260929 18:31:08.919426 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:08.921536 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6218658cc659415096f8b1a880ae7f03. Latest consensus state: current_term: 1 leader_uuid: "6218658cc659415096f8b1a880ae7f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } } }
I20260929 18:31:08.921741 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6218658cc659415096f8b1a880ae7f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6218658cc659415096f8b1a880ae7f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43679 } } }
I20260929 18:31:08.921897 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:08.921902 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03 [sys.catalog]: This master's current role is: LEADER
W20260929 18:31:08.924746 2187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:08.924823 2187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:08.924901 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:08.925782 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:08.931193 2188 catalog_manager.cc:1347] Generated new cluster ID: acfae86c38584aa2a2edf111a74aaadc
I20260929 18:31:08.931257 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:08.963330 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:08.964687 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:08.974344 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Generated new TSK 0
I20260929 18:31:08.975219 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:09.015394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43679
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:09.125675 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:09.125923 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:09.125970 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:09.129993 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:09.130048 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:09.130088 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:31:09.130138 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:09.130195 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:09.135273 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:09.136322 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:09.137303 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:09.146459 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:09.146529 2194 server_base.cc:1034] running on GCE node
W20260929 18:31:09.146456 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:09.146762 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:09.147188 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:09.147791 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:09.148995 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790706669148975 us; error 28 us; skew 500 ppm
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:09.152046 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:09.152930 2194 webserver.cc:460] Webserver started at http://127.0.98.1:33453/ using document root <none> and password file <none>
I20260929 18:31:09.153474 2194 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:09.153534 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:09.153738 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:09.155692 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4cba27025db4a07828e7b6bf4ce7d7a"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "8daaf60cf1e1f4d62d88b5f3be7ab88c"
server_key_iv: "88aeb0aadab94e84a16504d0e05f2556"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.156314 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4cba27025db4a07828e7b6bf4ce7d7a"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "8daaf60cf1e1f4d62d88b5f3be7ab88c"
server_key_iv: "88aeb0aadab94e84a16504d0e05f2556"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.159765 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:09.162256 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.163461 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:09.163609 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "f4cba27025db4a07828e7b6bf4ce7d7a"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "8daaf60cf1e1f4d62d88b5f3be7ab88c"
server_key_iv: "88aeb0aadab94e84a16504d0e05f2556"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.163724 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:09.179931 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:09.183293 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:09.183490 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:09.184967 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:09.185038 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.185108 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:09.185164 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:09.195801 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46085
I20260929 18:31:09.195914 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46085 every 8 connection(s)
I20260929 18:31:09.196985 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:09.199344 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.185735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33567 (local address 127.0.98.62:43679)
0929 18:31:09.185885 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.185890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.186747 (+ 857us) server_negotiation.cc:407] Connection header received
0929 18:31:09.187829 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.187836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.187909 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.187958 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.189397 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.190152 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.191729 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.192106 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.192247 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.195317 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.195322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.195328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.195361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.197204 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.197826 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.197829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.197830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.197904 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.198255 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.198257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.198258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.198432 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.198514 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.199114 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.199223 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:31:09.200099 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.186141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.1:33567)
0929 18:31:09.186586 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.186634 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.187562 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.188396 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.188402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.188712 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.189249 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.189262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.190282 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.190287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.191558 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.191571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.192333 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.192338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.192577 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.193210 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.193237 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.195144 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.197313 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.197317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.197327 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.197656 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.198025 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.198027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.198029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.198161 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.198563 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.198568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.198853 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.199045 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.199307 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20260929 18:31:09.201546 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43679
I20260929 18:31:09.201838 2325 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:09.201951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2194
I20260929 18:31:09.202065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:09.202423 392 external_mini_cluster.cc:1428] Setting key a780dc26dbcbdefc07a29fd9945092a6
I20260929 18:31:09.202574 2325 heartbeater.cc:511] Master 127.0.98.62:43679 requested a full tablet report, sending...
I20260929 18:31:09.204166 2122 ts_manager.cc:194] Registered new tserver with Master: f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085)
I20260929 18:31:09.205267 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33567
I20260929 18:31:09.211444 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.203764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55699 (local address 127.0.98.62:43679)
0929 18:31:09.203879 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.203883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.204303 (+ 420us) server_negotiation.cc:407] Connection header received
0929 18:31:09.204475 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.204478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.204525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.204580 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.205269 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.206006 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.206870 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.207083 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.207189 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.208328 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.208332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.208334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.208364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.209756 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.210294 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.210297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.210305 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.210383 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.210697 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.210699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.210701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.210867 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.210948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.211247 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.211340 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:31:09.211601 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.203697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.1:55699)
0929 18:31:09.204227 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.204243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.204367 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.204733 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.204738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.204975 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.205122 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.205132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.206203 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.206205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.206752 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.206758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.207195 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.207199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.207304 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.207804 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.207815 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.208133 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.209874 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.209879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.209882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.210163 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.210487 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.210491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.210497 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.210600 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.210964 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.210971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.211082 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.211211 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.211355 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:09.253233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43679
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:09.362401 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:09.362643 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:09.362686 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:09.366731 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:09.366784 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:09.366851 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:31:09.366881 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:09.366948 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:09.372192 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:09.373196 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:09.374176 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:09.383776 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:09.383842 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:09.383841 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:09.384094 2334 server_base.cc:1034] running on GCE node
I20260929 18:31:09.384547 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:09.385159 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:09.386369 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790706669386338 us; error 49 us; skew 500 ppm
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:09.389391 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:09.390232 2334 webserver.cc:460] Webserver started at http://127.0.98.2:33189/ using document root <none> and password file <none>
I20260929 18:31:09.390724 2334 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:09.390779 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:09.390929 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:09.393103 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "408d1ecaf5df4ee7856944591f8de9a6"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "9deaec3b6a03d8e87c2920514584d00a"
server_key_iv: "e9bad6a959c29171c8690a57409be31c"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.393677 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "408d1ecaf5df4ee7856944591f8de9a6"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "9deaec3b6a03d8e87c2920514584d00a"
server_key_iv: "e9bad6a959c29171c8690a57409be31c"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.397163 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:31:09.399531 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.400830 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:09.400970 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "408d1ecaf5df4ee7856944591f8de9a6"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "9deaec3b6a03d8e87c2920514584d00a"
server_key_iv: "e9bad6a959c29171c8690a57409be31c"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.401091 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:09.424839 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:09.428009 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:09.428263 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:09.429777 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:09.429847 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.429917 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:09.429970 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:09.440912 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45699
I20260929 18:31:09.441849 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:09.442631 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45699 every 8 connection(s)
I20260929 18:31:09.443583 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.430407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54211 (local address 127.0.98.62:43679)
0929 18:31:09.430547 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.430553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.431455 (+ 902us) server_negotiation.cc:407] Connection header received
0929 18:31:09.432621 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.432633 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.432700 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.432805 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.434112 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.434738 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.436072 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.436363 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.436480 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.439847 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.439851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.439854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.439879 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.441425 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.442011 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.442015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.442017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.442112 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.442571 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.442576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.442580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.442795 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.442897 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.443373 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.443469 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:31:09.444316 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.430857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.2:54211)
0929 18:31:09.431296 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.431344 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.432314 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.433055 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.433066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.433423 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.433957 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.433976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.434867 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.434871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.435951 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.435960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.436526 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.436530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.436737 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.437361 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.437378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.439663 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.441537 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.441542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.441554 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.441882 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.442281 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.442284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.442288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.442453 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.442904 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.442909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.443119 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.443279 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.443494 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
I20260929 18:31:09.445392 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43679
I20260929 18:31:09.445540 2465 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:09.446002 2465 heartbeater.cc:511] Master 127.0.98.62:43679 requested a full tablet report, sending...
I20260929 18:31:09.447062 2120 ts_manager.cc:194] Registered new tserver with Master: 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:09.447834 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54211
I20260929 18:31:09.449615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2334
I20260929 18:31:09.449731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:09.449981 392 external_mini_cluster.cc:1428] Setting key b7c0c6114029f2c256030a7b6faefa20
I20260929 18:31:09.454995 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.447468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.2:49403)
0929 18:31:09.447753 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.447767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.447864 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.448409 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.448411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.448562 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.448662 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.448667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.449388 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.449391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.450021 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.450029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.450569 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.450572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.450669 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.451172 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.451183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.451500 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.453478 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.453480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.453482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.453678 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.453976 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.453979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.453985 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.454063 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.454579 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.454582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.454667 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.454756 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.454856 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260929 18:31:09.455176 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.447588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49403 (local address 127.0.98.62:43679)
0929 18:31:09.447701 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.447705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.447854 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:31:09.447984 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.447987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.448055 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.448139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.448776 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.449266 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.450144 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.450438 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.450554 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.451639 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.451643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.451646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.451668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.453360 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.453777 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.453781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.453784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.453876 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.454157 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.454163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.454168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.454468 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.454573 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.454838 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.454967 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:09.475502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43679
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:09.582304 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:09.582553 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:09.582628 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:09.586622 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:09.586674 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:09.586714 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:31:09.586750 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:09.586819 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:09.591943 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:09.592952 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:09.593910 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:09.603660 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:09.603814 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:09.603782 2474 server_base.cc:1034] running on GCE node
W20260929 18:31:09.603932 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:09.604424 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:09.605106 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:09.606277 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790706669606246 us; error 42 us; skew 500 ppm
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:09.609041 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:09.609815 2474 webserver.cc:460] Webserver started at http://127.0.98.3:34653/ using document root <none> and password file <none>
I20260929 18:31:09.610345 2474 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:09.610422 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:09.610584 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:09.612529 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "fc9a982d3714d6ada75b5d625f17f288"
server_key_iv: "e3140fe9c8fe23226245c1a763cb36d4"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.613044 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "fc9a982d3714d6ada75b5d625f17f288"
server_key_iv: "e3140fe9c8fe23226245c1a763cb36d4"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.616460 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:31:09.618719 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.619972 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:09.620127 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
format_stamp: "Formatted at 2026-09-29 18:31:09 on dist-test-slave-fmv8"
server_key: "fc9a982d3714d6ada75b5d625f17f288"
server_key_iv: "e3140fe9c8fe23226245c1a763cb36d4"
server_key_version: "encryptionkey@0"
I20260929 18:31:09.620267 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:09.646066 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:09.648931 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:09.649075 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:09.650357 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:09.650431 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:09.650506 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:09.650557 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:09.661522 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40267
I20260929 18:31:09.661552 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40267 every 8 connection(s)
I20260929 18:31:09.662878 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:09.665566 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.651244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59191 (local address 127.0.98.62:43679)
0929 18:31:09.651373 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.651377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.652408 (+ 1031us) server_negotiation.cc:407] Connection header received
0929 18:31:09.653538 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.653542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.653641 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.653761 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.655543 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.656470 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.658291 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.658634 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.658769 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.661805 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.661810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.661814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.661839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.663451 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.664071 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.664074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.664076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.664205 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.664529 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.664534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.664535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.664748 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.664836 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.665286 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.665403 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:31:09.666410 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.651768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.3:59191)
0929 18:31:09.652263 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.652303 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.653267 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.654203 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.654212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.654653 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.655384 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.655405 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.656603 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.656607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.658156 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.658166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.658807 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.658810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.658944 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.659634 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.659652 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.661636 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.663567 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.663573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.663588 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.663960 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.664296 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.664299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.664301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.664434 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.664836 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.664843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.665068 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.665245 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.665534 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260929 18:31:09.667625 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43679
I20260929 18:31:09.667764 2605 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:09.668288 2605 heartbeater.cc:511] Master 127.0.98.62:43679 requested a full tablet report, sending...
I20260929 18:31:09.669255 2122 ts_manager.cc:194] Registered new tserver with Master: 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267)
I20260929 18:31:09.669926 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59191
I20260929 18:31:09.671407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2474
I20260929 18:31:09.671528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:09.671777 392 external_mini_cluster.cc:1428] Setting key d6b0b2071d3efc878d717748753dd8a2
I20260929 18:31:09.673791 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:31:09.676822 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.669817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54301 (local address 127.0.98.62:43679)
0929 18:31:09.669930 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.669934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.670075 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:31:09.670215 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.670218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.670254 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.670355 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.670783 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.671356 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.672366 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.672680 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.672780 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.673920 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.673924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.673926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.673947 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.675175 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.675676 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.675679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.675680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.675751 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.676041 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.676043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.676044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.676265 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.676340 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.676634 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.676718 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:09.676923 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.669709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.98.3:54301)
0929 18:31:09.669995 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.670010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.670135 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.670367 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.670370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.670570 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.670670 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.670677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.671498 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.671502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.672258 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.672271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.672788 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.672790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.672906 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.673369 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.673383 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.673782 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.675282 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.675287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.675289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.675566 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.675847 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.675852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.675855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.675954 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.676355 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.676357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.676459 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.676585 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.676733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:09.698520 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.691114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.98.62:43679)
0929 18:31:09.691237 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.691240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.691253 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:09.691286 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.691288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.691329 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.691398 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.692009 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.692576 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.693426 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.693619 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.693690 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.695831 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.695835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.695836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.695856 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.697088 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.697472 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.697475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.697477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.697551 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.697798 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.697800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.697801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.697952 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.698098 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.698324 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.698412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:31:09.701144 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:09.704373 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:09.718503 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.713564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.2:45699)
0929 18:31:09.713837 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.713847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.714399 (+ 552us) server_negotiation.cc:407] Connection header received
0929 18:31:09.714640 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.714646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.714809 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.714951 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:09.715148 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.716035 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.717411 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.717955 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.718079 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.718241 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.718294 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.718378 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
I20260929 18:31:09.718487 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.713199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45699 (local address 127.0.0.1:45678)
0929 18:31:09.714312 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.714324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.714466 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.714940 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.714943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.714961 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:09.715038 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.715043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.716159 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.716191 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.717301 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.717307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.718092 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.718095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.718175 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.718196 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.718269 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.718337 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":10,"thread_start_us":91,"threads_started":1}
I20260929 18:31:09.718742 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.713130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.98.1:46085)
0929 18:31:09.713654 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.713662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.714074 (+ 412us) server_negotiation.cc:407] Connection header received
0929 18:31:09.714208 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.714214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.714403 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.714547 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:09.714833 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.715772 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.717251 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.717991 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.718088 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.718231 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.718527 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.718623 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":86,"threads_started":1}
I20260929 18:31:09.718762 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.712649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.3:40267)
0929 18:31:09.713329 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.713342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.713366 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:09.713445 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.713453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.713668 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.713824 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:09.714236 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.715405 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.716920 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.717603 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.717717 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.717869 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.718552 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.718642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":104,"threads_started":1}
I20260929 18:31:09.718497 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.712492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40267 (local address 127.0.0.1:41956)
0929 18:31:09.712864 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.712914 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.713190 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.714001 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.714005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.714024 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:09.714115 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.714119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.715517 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.715520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.716806 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.716815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.717930 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.717934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.718048 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.718094 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.718193 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.718317 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260929 18:31:09.719653 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.712879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46085 (local address 127.0.0.1:36958)
0929 18:31:09.713985 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.714000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.714096 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.714610 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.714613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.714626 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:09.714717 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.714723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.715893 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.715896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.717128 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.717137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.718285 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.718288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.718376 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.718397 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.719449 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.719526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 18:31:09.721946 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 (DEFAULT_TABLE table=test-table [id=19d96804525641fda82d8c0918db0bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:09.722203 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 (DEFAULT_TABLE table=test-table [id=19d96804525641fda82d8c0918db0bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:09.722283 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 (DEFAULT_TABLE table=test-table [id=19d96804525641fda82d8c0918db0bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:09.723078 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8308eec395284ad8b560a1b75eb7daa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.723074 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8308eec395284ad8b560a1b75eb7daa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.723075 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8308eec395284ad8b560a1b75eb7daa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.729807 2628 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Bootstrap starting.
I20260929 18:31:09.730626 2630 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Bootstrap starting.
I20260929 18:31:09.731086 2629 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Bootstrap starting.
I20260929 18:31:09.731992 2628 tablet_bootstrap.cc:654] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.732688 2630 tablet_bootstrap.cc:654] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.732787 2628 log.cc:824] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:09.733011 2629 tablet_bootstrap.cc:654] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.733513 2630 log.cc:824] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:09.733731 2629 log.cc:824] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:09.734849 2628 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: No bootstrap required, opened a new log
I20260929 18:31:09.735026 2628 ts_tablet_manager.cc:1397] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:31:09.735544 2630 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: No bootstrap required, opened a new log
I20260929 18:31:09.735575 2629 tablet_bootstrap.cc:492] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: No bootstrap required, opened a new log
I20260929 18:31:09.735864 2629 ts_tablet_manager.cc:1397] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:09.735852 2630 ts_tablet_manager.cc:1397] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:09.738178 2628 raft_consensus.cc:348] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.738409 2628 raft_consensus.cc:374] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.738469 2628 raft_consensus.cc:729] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1ecaf5df4ee7856944591f8de9a6, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.738914 2628 consensus_queue.cc:260] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.739552 2465 heartbeater.cc:503] Master 127.0.98.62:43679 was elected leader, sending a full tablet report...
I20260929 18:31:09.739867 2628 ts_tablet_manager.cc:1428] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:31:09.740067 2629 raft_consensus.cc:348] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.740342 2629 raft_consensus.cc:374] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.740404 2629 raft_consensus.cc:729] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.740846 2629 consensus_queue.cc:260] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.740875 2630 raft_consensus.cc:348] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.741098 2630 raft_consensus.cc:374] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.741161 2630 raft_consensus.cc:729] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.741546 2630 consensus_queue.cc:260] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.741843 2629 ts_tablet_manager.cc:1428] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:31:09.742157 2325 heartbeater.cc:503] Master 127.0.98.62:43679 was elected leader, sending a full tablet report...
I20260929 18:31:09.743359 2630 ts_tablet_manager.cc:1428] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:31:09.743690 2605 heartbeater.cc:503] Master 127.0.98.62:43679 was elected leader, sending a full tablet report...
I20260929 18:31:09.751993 2635 raft_consensus.cc:482] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:09.752138 2635 raft_consensus.cc:504] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.753218 2635 leader_election.cc:288] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:09.757371 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.753459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40267 (local address 127.0.98.1:43563)
0929 18:31:09.753814 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.753826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.753951 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.754209 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.754212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.754228 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:09.754375 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.754382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.755223 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.755226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.756298 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.756307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.756930 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.757032 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.757060 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.757134 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.757205 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":26,"thread_start_us":140,"threads_started":1}
I20260929 18:31:09.757429 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.753901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45699 (local address 127.0.98.1:39783)
0929 18:31:09.754202 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.754214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.754304 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.754544 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.754547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.754567 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:09.754673 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.754679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.755482 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.755484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.756298 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.756306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.756971 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.757112 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.757137 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.757213 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.757278 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":19,"thread_start_us":95,"threads_started":1}
I20260929 18:31:09.757439 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.754002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39783 (local address 127.0.98.2:45699)
0929 18:31:09.754128 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.754133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.754295 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:31:09.754384 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.754388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.754445 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.754524 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:09.754805 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.755370 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756432 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.756832 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756950 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.757010 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.757248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.757318 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:09.757414 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.753529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43563 (local address 127.0.98.3:40267)
0929 18:31:09.753696 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.753701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.753909 (+ 208us) server_negotiation.cc:407] Connection header received
0929 18:31:09.754062 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.754065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.754116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.754194 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:09.754506 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.755123 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756428 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.756815 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.756894 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.756984 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.757234 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.757299 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:31:09.758145 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8308eec395284ad8b560a1b75eb7daa5" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6" is_pre_election: true
I20260929 18:31:09.758263 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8308eec395284ad8b560a1b75eb7daa5" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" is_pre_election: true
I20260929 18:31:09.758507 2419 raft_consensus.cc:2454] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:09.758540 2559 raft_consensus.cc:2454] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:09.758967 2211 leader_election.cc:302] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408d1ecaf5df4ee7856944591f8de9a6, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:09.759255 2635 raft_consensus.cc:2781] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:09.759351 2635 raft_consensus.cc:482] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:09.759410 2635 raft_consensus.cc:3037] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:09.760434 2635 raft_consensus.cc:504] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.760813 2635 leader_election.cc:288] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Requested vote from peers 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:09.761184 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8308eec395284ad8b560a1b75eb7daa5" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
I20260929 18:31:09.761317 2559 raft_consensus.cc:3037] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:09.762315 2559 raft_consensus.cc:2454] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:09.762600 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8308eec395284ad8b560a1b75eb7daa5" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6"
I20260929 18:31:09.762665 2211 leader_election.cc:302] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:09.762737 2419 raft_consensus.cc:3037] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:09.763662 2419 raft_consensus.cc:2454] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:09.763988 2635 raft_consensus.cc:2781] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:09.764253 2635 raft_consensus.cc:686] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Becoming Leader. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Running, Role: LEADER
I20260929 18:31:09.764573 2635 consensus_queue.cc:237] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } }
I20260929 18:31:09.768018 2122 catalog_manager.cc:5495] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } health_report { overall_health: HEALTHY } } }
Sep 29 18:31:09 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:09.830579 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.822018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43679 (local address 127.0.0.1:40266)
0929 18:31:09.822417 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:09.822430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:09.822531 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:09.822810 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:09.822816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:09.823278 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:09.823371 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.823377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.824137 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.824140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.824804 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:09.824813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.825272 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.825275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:09.825353 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.825895 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:09.825907 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:09.827345 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:09.829140 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.829148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.829152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.829369 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.829695 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:09.829697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:09.829699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:09.829781 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:09.830121 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:09.830127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:09.830237 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:31:09.830329 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:31:09.830435 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":228,"threads_started":1}
I20260929 18:31:09.830653 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:09.822095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.0.98.62:43679)
0929 18:31:09.822345 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:09.822349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:09.822510 (+ 161us) server_negotiation.cc:407] Connection header received
0929 18:31:09.822614 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:09.822616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:09.822669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:09.822737 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:09.823505 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.824020 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.824942 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:09.825147 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:09.825235 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:09.827469 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:09.827472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:09.827474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:09.827493 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:09.829033 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.829491 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.829494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.829496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.829571 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:09.829868 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:09.829870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:09.829871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:09.830031 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:09.830106 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:09.830412 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:31:09.830501 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260929 18:31:09.834509 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40266:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:31:09.835175 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:09.839854 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 613122f69efd4a5c92bed35213029bbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68490c2a4b8444bf87842c45373f540c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:09.840150 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 613122f69efd4a5c92bed35213029bbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68490c2a4b8444bf87842c45373f540c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:09.840175 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 613122f69efd4a5c92bed35213029bbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68490c2a4b8444bf87842c45373f540c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:09.840709 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613122f69efd4a5c92bed35213029bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.840824 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613122f69efd4a5c92bed35213029bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.840830 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613122f69efd4a5c92bed35213029bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:09.843159 2629 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: Bootstrap starting.
I20260929 18:31:09.843508 2628 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: Bootstrap starting.
I20260929 18:31:09.843636 2630 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: Bootstrap starting.
I20260929 18:31:09.844244 2629 tablet_bootstrap.cc:654] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.844502 2628 tablet_bootstrap.cc:654] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.844580 2630 tablet_bootstrap.cc:654] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:09.845746 2630 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: No bootstrap required, opened a new log
I20260929 18:31:09.845755 2628 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: No bootstrap required, opened a new log
I20260929 18:31:09.845813 2629 tablet_bootstrap.cc:492] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: No bootstrap required, opened a new log
I20260929 18:31:09.845829 2630 ts_tablet_manager.cc:1397] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:09.845829 2628 ts_tablet_manager.cc:1397] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:09.845889 2629 ts_tablet_manager.cc:1397] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:31:09.846262 2630 raft_consensus.cc:348] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846262 2628 raft_consensus.cc:348] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846376 2630 raft_consensus.cc:374] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.846376 2628 raft_consensus.cc:374] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.846411 2630 raft_consensus.cc:729] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.846411 2628 raft_consensus.cc:729] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1ecaf5df4ee7856944591f8de9a6, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.846431 2629 raft_consensus.cc:348] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846549 2629 raft_consensus.cc:374] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:09.846530 2630 consensus_queue.cc:260] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846530 2628 consensus_queue.cc:260] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846590 2629 raft_consensus.cc:729] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Initialized, Role: FOLLOWER
I20260929 18:31:09.846755 2629 consensus_queue.cc:260] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:09.846791 2630 ts_tablet_manager.cc:1428] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:09.846791 2628 ts_tablet_manager.cc:1428] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:09.846992 2635 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:09.847050 2634 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:09.847122 2635 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:09.847168 2634 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:09.847332 2629 ts_tablet_manager.cc:1428] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 18:31:09.847334 2636 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:09.847462 2636 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 18:31:09.913826 2606 tablet.cc:2369] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:09.943051 2466 tablet.cc:2369] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:31:09.948325 2326 tablet.cc:2369] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:10.046593 2636 raft_consensus.cc:482] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:10.046761 2636 raft_consensus.cc:504] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.048041 2636 leader_election.cc:288] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:10.052150 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.048385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45699 (local address 127.0.98.3:38393)
0929 18:31:10.048513 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:10.048529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:10.048643 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:10.049037 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:10.049042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:10.049065 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:10.049197 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.049203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.050155 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.050159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.050988 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.050997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.051683 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.051834 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.051859 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:31:10.051938 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:31:10.051993 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:31:10.052211 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.048538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38393 (local address 127.0.98.2:45699)
0929 18:31:10.048684 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.048688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.048735 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:31:10.048806 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.048810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.048862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.048964 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:10.049407 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.050037 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051121 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.051560 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051650 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.051701 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.051965 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.052055 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:31:10.052312 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.048383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58963 (local address 127.0.98.1:46085)
0929 18:31:10.048517 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.048524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.049054 (+ 530us) server_negotiation.cc:407] Connection header received
0929 18:31:10.049147 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.049150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.049202 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.049322 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:10.049656 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.050174 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051286 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.051704 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051813 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.051871 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.052086 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.052177 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:31:10.052567 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.048395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46085 (local address 127.0.98.3:58963)
0929 18:31:10.048931 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:10.048947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:10.049042 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:10.049398 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:10.049401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:10.049420 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:10.049538 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.049544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.050283 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.050286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.051157 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.051166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.051833 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.051836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.051951 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.051973 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:31:10.052047 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:31:10.052114 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":146,"thread_start_us":130,"threads_started":1}
I20260929 18:31:10.052824 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613122f69efd4a5c92bed35213029bbc" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6" is_pre_election: true
I20260929 18:31:10.052987 2419 raft_consensus.cc:2454] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 0.
I20260929 18:31:10.053193 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613122f69efd4a5c92bed35213029bbc" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" is_pre_election: true
I20260929 18:31:10.053371 2279 raft_consensus.cc:2454] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 0.
I20260929 18:31:10.053368 2491 leader_election.cc:302] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, 408d1ecaf5df4ee7856944591f8de9a6; no voters:
I20260929 18:31:10.053682 2636 raft_consensus.cc:2781] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:10.053747 2636 raft_consensus.cc:482] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:10.053802 2636 raft_consensus.cc:3037] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.054744 2636 raft_consensus.cc:504] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.055094 2636 leader_election.cc:288] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 election: Requested vote from peers f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:10.055374 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613122f69efd4a5c92bed35213029bbc" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a"
I20260929 18:31:10.055399 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613122f69efd4a5c92bed35213029bbc" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6"
I20260929 18:31:10.055500 2279 raft_consensus.cc:3037] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.055500 2419 raft_consensus.cc:3037] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.056655 2279 raft_consensus.cc:2454] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 1.
I20260929 18:31:10.056655 2419 raft_consensus.cc:2454] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 1.
I20260929 18:31:10.056946 2493 leader_election.cc:302] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:10.057132 2636 raft_consensus.cc:2781] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:10.057363 2636 raft_consensus.cc:686] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Running, Role: LEADER
I20260929 18:31:10.057996 2636 consensus_queue.cc:237] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.060086 2651 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: TxnStatusTablet state changed. Reason: New leader 3dd70bb0595a4398a367d9ad6b3e5f71. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.060227 2651 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:10.060525 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:31:10.060729 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:31:10.061492 2120 catalog_manager.cc:5495] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } health_report { overall_health: HEALTHY } } }
I20260929 18:31:10.086551 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.082351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40267 (local address 127.0.0.1:41964)
0929 18:31:10.082576 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:10.082600 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:10.082735 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:10.083069 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:10.083071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:10.083078 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:10.083176 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.083181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.084253 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.084256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.084864 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.084872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.085443 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.085445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.085512 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.085553 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.086303 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.086315 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 18:31:10.086370 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:31:10.086441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 18:31:10.086593 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.082504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.98.3:40267)
0929 18:31:10.082626 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.082630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.082679 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:31:10.082875 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.082878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.082942 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.083043 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:10.083282 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.084063 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.084981 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.085332 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.085438 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.085682 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.086206 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.086321 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.086382 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.086438 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:31:10.091519 2279 raft_consensus.cc:1261] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Refusing update from remote peer 3dd70bb0595a4398a367d9ad6b3e5f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.091552 2419 raft_consensus.cc:1261] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Refusing update from remote peer 3dd70bb0595a4398a367d9ad6b3e5f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.092137 2651 consensus_queue.cc:1035] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:10.092340 2636 consensus_queue.cc:1035] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:10.095480 2635 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: TxnStatusTablet state changed. Reason: New leader 3dd70bb0595a4398a367d9ad6b3e5f71. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.095602 2635 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:10.095724 2634 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: TxnStatusTablet state changed. Reason: New leader 3dd70bb0595a4398a367d9ad6b3e5f71. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.095844 2634 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:10.098906 2651 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.099027 2651 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:10.099124 2651 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.099211 2651 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 3dd70bb0595a4398a367d9ad6b3e5f71: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:10.099349 2635 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.099448 2635 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P f4cba27025db4a07828e7b6bf4ce7d7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:10.099713 2634 tablet_replica.cc:439] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } } }
I20260929 18:31:10.099817 2634 tablet_replica.cc:442] T 613122f69efd4a5c92bed35213029bbc P 408d1ecaf5df4ee7856944591f8de9a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:10.100070 2656 mvcc.cc:205] Tried to move back new op lower bound from 7334734520688152576 to 7334734520560111616. Current Snapshot: MvccSnapshot[applied={T|T < 7334734520688152576}]
I20260929 18:31:10.100366 2657 mvcc.cc:205] Tried to move back new op lower bound from 7334734520688152576 to 7334734520560111616. Current Snapshot: MvccSnapshot[applied={T|T < 7334734520688152576}]
I20260929 18:31:10.100770 2658 mvcc.cc:205] Tried to move back new op lower bound from 7334734520688152576 to 7334734520560111616. Current Snapshot: MvccSnapshot[applied={T|T < 7334734520688152576}]
I20260929 18:31:10.110435 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.106483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.0.98.1:46085)
0929 18:31:10.106596 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.106606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.106664 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:31:10.106808 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.106810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.106854 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.106932 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:10.107234 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.107733 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.108968 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.109207 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.109289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.109639 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.110028 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.110123 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.110230 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.110277 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:31:10.121444 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.117560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40267 (local address 127.0.98.1:47525)
0929 18:31:10.117873 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:10.117888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:10.118020 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:10.118311 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:10.118313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:10.118321 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:10.118453 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.118459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.119213 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.119217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.119990 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.119999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.120632 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.120636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.120737 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.120786 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.121152 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.121157 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:31:10.121208 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:31:10.121291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260929 18:31:10.121446 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.117729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47525 (local address 127.0.98.3:40267)
0929 18:31:10.117860 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.117864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.117971 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:31:10.118136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.118140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.118217 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.118301 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:10.118571 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.119082 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.120113 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.120499 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.120634 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.120966 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.121050 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.121130 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.121242 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.121314 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:31:10.128732 2419 raft_consensus.cc:1261] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.128958 2559 raft_consensus.cc:1261] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.129211 2635 consensus_queue.cc:1035] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:10.129563 2635 consensus_queue.cc:1035] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:10.151216 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.147833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46085 (local address 127.0.98.3:36625)
0929 18:31:10.147990 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:10.148004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:10.148085 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:10.148344 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:10.148348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:10.148356 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:10.148468 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.148474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.149410 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.149414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.150030 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:10.150039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.150508 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.150511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:10.150604 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.150655 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.150953 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.150961 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:31:10.151024 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:31:10.151099 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:31:10.151207 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:10.147910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36625 (local address 127.0.98.1:46085)
0929 18:31:10.148023 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:10.148027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:10.148049 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:31:10.148194 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:10.148198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:10.148244 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:10.148315 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:10.148567 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.149320 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.150158 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:10.150398 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:10.150508 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:10.150787 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:10.150867 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:10.150947 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:10.151032 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 18:31:10.151078 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:31:10.175351 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:10.175557 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:10.178325 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Sending DeleteTablet for 3 replicas of tablet 8308eec395284ad8b560a1b75eb7daa5
I20260929 18:31:10.179138 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41956
I20260929 18:31:10.179185 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36958
I20260929 18:31:10.179993 2676 tablet_replica.cc:330] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: stopping tablet replica
I20260929 18:31:10.179975 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 8308eec395284ad8b560a1b75eb7daa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45678
I20260929 18:31:10.180356 2676 raft_consensus.cc:2229] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:10.180774 2676 raft_consensus.cc:2258] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:10.181187 2677 tablet_replica.cc:330] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: stopping tablet replica
I20260929 18:31:10.181474 2677 raft_consensus.cc:2229] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:10.181670 2677 raft_consensus.cc:2258] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:10.181596 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:10.182051 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:10.183075 2677 ts_tablet_manager.cc:1905] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:10.186698 2677 ts_tablet_manager.cc:1918] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:10.186810 2677 log.cc:1196] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/8308eec395284ad8b560a1b75eb7daa5
I20260929 18:31:10.187105 2677 ts_tablet_manager.cc:1939] T 8308eec395284ad8b560a1b75eb7daa5 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting consensus metadata
I20260929 18:31:10.187574 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a (DEFAULT_TABLE table=test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:10.187844 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a6743875934fcf9f641c4bb4d42c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:10.189546 2676 ts_tablet_manager.cc:1905] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:10.190249 2629 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Bootstrap starting.
I20260929 18:31:10.191159 2629 tablet_bootstrap.cc:654] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:10.192173 2107 catalog_manager.cc:4841] TS 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699): tablet 8308eec395284ad8b560a1b75eb7daa5 (table test-table [id=19d96804525641fda82d8c0918db0bf0]) successfully deleted
I20260929 18:31:10.192478 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a (DEFAULT_TABLE table=test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:10.192751 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a6743875934fcf9f641c4bb4d42c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:10.193038 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a (DEFAULT_TABLE table=test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:10.193289 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a6743875934fcf9f641c4bb4d42c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:10.195020 2628 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Bootstrap starting.
I20260929 18:31:10.195920 2628 tablet_bootstrap.cc:654] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:10.196939 2630 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Bootstrap starting.
I20260929 18:31:10.197311 2676 ts_tablet_manager.cc:1918] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:10.197391 2676 log.cc:1196] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/8308eec395284ad8b560a1b75eb7daa5
I20260929 18:31:10.197676 2676 ts_tablet_manager.cc:1939] T 8308eec395284ad8b560a1b75eb7daa5 P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting consensus metadata
I20260929 18:31:10.197831 2630 tablet_bootstrap.cc:654] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:10.198745 2109 catalog_manager.cc:4841] TS f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085): tablet 8308eec395284ad8b560a1b75eb7daa5 (table test-table [id=19d96804525641fda82d8c0918db0bf0]) successfully deleted
I20260929 18:31:10.199592 2628 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: No bootstrap required, opened a new log
I20260929 18:31:10.199680 2628 ts_tablet_manager.cc:1397] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:31:10.200088 2628 raft_consensus.cc:348] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.200220 2629 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: No bootstrap required, opened a new log
I20260929 18:31:10.200227 2628 raft_consensus.cc:374] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:10.200260 2628 raft_consensus.cc:729] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1ecaf5df4ee7856944591f8de9a6, State: Initialized, Role: FOLLOWER
I20260929 18:31:10.200297 2629 ts_tablet_manager.cc:1397] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260929 18:31:10.200385 2628 consensus_queue.cc:260] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.200610 2628 ts_tablet_manager.cc:1428] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:10.200697 2629 raft_consensus.cc:348] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.200821 2629 raft_consensus.cc:374] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:10.200855 2629 raft_consensus.cc:729] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Initialized, Role: FOLLOWER
I20260929 18:31:10.200964 2629 consensus_queue.cc:260] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.201226 2629 ts_tablet_manager.cc:1428] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:10.203758 2630 tablet_bootstrap.cc:492] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: No bootstrap required, opened a new log
I20260929 18:31:10.203860 2630 ts_tablet_manager.cc:1397] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 18:31:10.204414 2630 raft_consensus.cc:348] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.204524 2630 raft_consensus.cc:374] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:10.204553 2630 raft_consensus.cc:729] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Initialized, Role: FOLLOWER
I20260929 18:31:10.204687 2630 consensus_queue.cc:260] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.204926 2630 ts_tablet_manager.cc:1428] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:10.205958 2675 tablet_replica.cc:330] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: stopping tablet replica
I20260929 18:31:10.206275 2675 raft_consensus.cc:2229] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:10.206467 2675 raft_consensus.cc:2258] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:10.209398 2675 ts_tablet_manager.cc:1905] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:10.212493 2675 ts_tablet_manager.cc:1918] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:10.212682 2675 log.cc:1196] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/8308eec395284ad8b560a1b75eb7daa5
I20260929 18:31:10.213073 2675 ts_tablet_manager.cc:1939] T 8308eec395284ad8b560a1b75eb7daa5 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting consensus metadata
I20260929 18:31:10.214323 2107 catalog_manager.cc:4841] TS 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267): tablet 8308eec395284ad8b560a1b75eb7daa5 (table test-table [id=19d96804525641fda82d8c0918db0bf0]) successfully deleted
I20260929 18:31:10.465327 2635 raft_consensus.cc:482] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:10.465468 2635 raft_consensus.cc:504] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.466125 2635 leader_election.cc:288] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699), 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267)
I20260929 18:31:10.466724 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a6743875934fcf9f641c4bb4d42c6a" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" is_pre_election: true
I20260929 18:31:10.466724 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a6743875934fcf9f641c4bb4d42c6a" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6" is_pre_election: true
I20260929 18:31:10.466877 2559 raft_consensus.cc:2454] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:10.466877 2419 raft_consensus.cc:2454] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:10.467217 2211 leader_election.cc:302] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:10.467391 2635 raft_consensus.cc:2781] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:10.467451 2635 raft_consensus.cc:482] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:10.467494 2635 raft_consensus.cc:3037] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.468743 2635 raft_consensus.cc:504] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.469061 2635 leader_election.cc:288] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Requested vote from peers 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699), 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267)
I20260929 18:31:10.469466 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a6743875934fcf9f641c4bb4d42c6a" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6"
I20260929 18:31:10.469466 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a6743875934fcf9f641c4bb4d42c6a" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
I20260929 18:31:10.469591 2419 raft_consensus.cc:3037] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.469591 2559 raft_consensus.cc:3037] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:10.470680 2559 raft_consensus.cc:2454] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:10.470680 2419 raft_consensus.cc:2454] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:10.470979 2211 leader_election.cc:302] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:10.471163 2635 raft_consensus.cc:2781] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:10.471259 2635 raft_consensus.cc:686] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Becoming Leader. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Running, Role: LEADER
I20260929 18:31:10.471400 2635 consensus_queue.cc:237] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:10.473096 2120 catalog_manager.cc:5495] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:10.507773 2559 raft_consensus.cc:1261] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.507773 2419 raft_consensus.cc:1261] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:10.508193 2635 consensus_queue.cc:1035] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:10.508323 2671 consensus_queue.cc:1035] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:31:10.522580 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:31:12.073272 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:12.073419 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:12.075646 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Sending DeleteTablet for 3 replicas of tablet 49a6743875934fcf9f641c4bb4d42c6a
I20260929 18:31:12.076210 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45678
I20260929 18:31:12.076246 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41956
I20260929 18:31:12.076227 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a6743875934fcf9f641c4bb4d42c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36958
I20260929 18:31:12.076665 2706 tablet_replica.cc:330] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: stopping tablet replica
I20260929 18:31:12.076803 2706 raft_consensus.cc:2229] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:12.076972 2706 raft_consensus.cc:2258] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.077636 2706 ts_tablet_manager.cc:1905] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.078055 2705 tablet_replica.cc:330] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: stopping tablet replica
I20260929 18:31:12.078209 2705 raft_consensus.cc:2229] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:12.078331 2705 raft_consensus.cc:2258] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.078557 2707 tablet_replica.cc:330] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: stopping tablet replica
I20260929 18:31:12.078919 2705 ts_tablet_manager.cc:1905] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.079073 2707 raft_consensus.cc:2229] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:12.079298 2707 raft_consensus.cc:2258] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.079972 2707 ts_tablet_manager.cc:1905] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.081164 2706 ts_tablet_manager.cc:1918] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:31:12.081244 2706 log.cc:1196] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/49a6743875934fcf9f641c4bb4d42c6a
I20260929 18:31:12.081562 2706 ts_tablet_manager.cc:1939] T 49a6743875934fcf9f641c4bb4d42c6a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting consensus metadata
I20260929 18:31:12.081734 2705 ts_tablet_manager.cc:1918] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:31:12.081822 2705 log.cc:1196] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/49a6743875934fcf9f641c4bb4d42c6a
I20260929 18:31:12.082129 2705 ts_tablet_manager.cc:1939] T 49a6743875934fcf9f641c4bb4d42c6a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting consensus metadata
I20260929 18:31:12.082340 2107 catalog_manager.cc:4841] TS 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267): tablet 49a6743875934fcf9f641c4bb4d42c6a (table test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]) successfully deleted
I20260929 18:31:12.082705 2707 ts_tablet_manager.cc:1918] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:31:12.082801 2707 log.cc:1196] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/49a6743875934fcf9f641c4bb4d42c6a
I20260929 18:31:12.082825 2107 catalog_manager.cc:4841] TS 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699): tablet 49a6743875934fcf9f641c4bb4d42c6a (table test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]) successfully deleted
I20260929 18:31:12.083078 2707 ts_tablet_manager.cc:1939] T 49a6743875934fcf9f641c4bb4d42c6a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting consensus metadata
I20260929 18:31:12.083608 2109 catalog_manager.cc:4841] TS f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085): tablet 49a6743875934fcf9f641c4bb4d42c6a (table test-table [id=c0f2146de09b428ca8d97645dd9dbcb1]) successfully deleted
Sep 29 18:31:12 dist-test-slave-fmv8 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:31:12 dist-test-slave-fmv8 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:12.109790 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:12.097807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.98.62:43679)
0929 18:31:12.098084 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:12.098097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:12.098186 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:31:12.098361 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:12.098364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:12.098417 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:12.098516 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:12.099215 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.099757 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.100692 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.100927 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.101035 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:12.106753 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:12.106756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:12.106758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:12.106783 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:12.108103 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:12.108555 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:12.108561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:12.108566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:12.108652 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:12.108919 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:12.108923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:12.108925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:12.109115 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:12.109259 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:12.109487 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:31:12.109613 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260929 18:31:12.112444 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:12.112912 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:12.117547 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a (DEFAULT_TABLE table=test-table [id=4afd6a5ca5394e76b2ee569685f785d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.117568 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a (DEFAULT_TABLE table=test-table [id=4afd6a5ca5394e76b2ee569685f785d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.117679 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a (DEFAULT_TABLE table=test-table [id=4afd6a5ca5394e76b2ee569685f785d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.117851 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29c4f4eba4442e8eb13e89f466678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.117862 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29c4f4eba4442e8eb13e89f466678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.117911 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29c4f4eba4442e8eb13e89f466678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.119904 2716 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Bootstrap starting.
I20260929 18:31:12.120301 2717 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Bootstrap starting.
I20260929 18:31:12.120828 2718 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Bootstrap starting.
I20260929 18:31:12.121157 2716 tablet_bootstrap.cc:654] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.121157 2717 tablet_bootstrap.cc:654] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.121654 2718 tablet_bootstrap.cc:654] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.122427 2717 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: No bootstrap required, opened a new log
I20260929 18:31:12.122522 2717 ts_tablet_manager.cc:1397] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:31:12.122802 2716 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: No bootstrap required, opened a new log
I20260929 18:31:12.122892 2716 ts_tablet_manager.cc:1397] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:12.122954 2717 raft_consensus.cc:348] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.123098 2717 raft_consensus.cc:374] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.123139 2717 raft_consensus.cc:729] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.123262 2717 consensus_queue.cc:260] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.123342 2716 raft_consensus.cc:348] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.123456 2716 raft_consensus.cc:374] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.123489 2716 raft_consensus.cc:729] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1ecaf5df4ee7856944591f8de9a6, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.123551 2717 ts_tablet_manager.cc:1428] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:31:12.123605 2716 consensus_queue.cc:260] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.124044 2716 ts_tablet_manager.cc:1428] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:12.124722 2718 tablet_bootstrap.cc:492] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: No bootstrap required, opened a new log
I20260929 18:31:12.124854 2718 ts_tablet_manager.cc:1397] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:31:12.125368 2718 raft_consensus.cc:348] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.125487 2718 raft_consensus.cc:374] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.125557 2718 raft_consensus.cc:729] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.125710 2718 consensus_queue.cc:260] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.126029 2718 ts_tablet_manager.cc:1428] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:12.141657 2691 raft_consensus.cc:482] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:12.141772 2691 raft_consensus.cc:504] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.142098 2691 leader_election.cc:288] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:12.142504 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29c4f4eba4442e8eb13e89f466678a" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6" is_pre_election: true
I20260929 18:31:12.142508 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29c4f4eba4442e8eb13e89f466678a" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" is_pre_election: true
I20260929 18:31:12.142632 2419 raft_consensus.cc:2454] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 0.
I20260929 18:31:12.142661 2279 raft_consensus.cc:2454] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 0.
I20260929 18:31:12.142968 2491 leader_election.cc:302] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, 408d1ecaf5df4ee7856944591f8de9a6; no voters:
I20260929 18:31:12.143172 2691 raft_consensus.cc:2781] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:12.143254 2691 raft_consensus.cc:482] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:12.143306 2691 raft_consensus.cc:3037] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.144411 2691 raft_consensus.cc:504] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.144773 2691 leader_election.cc:288] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 election: Requested vote from peers f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085), 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699)
I20260929 18:31:12.145169 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29c4f4eba4442e8eb13e89f466678a" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a"
I20260929 18:31:12.145169 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29c4f4eba4442e8eb13e89f466678a" candidate_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6"
I20260929 18:31:12.145290 2279 raft_consensus.cc:3037] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.145290 2419 raft_consensus.cc:3037] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.146414 2279 raft_consensus.cc:2454] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 1.
I20260929 18:31:12.146435 2419 raft_consensus.cc:2454] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd70bb0595a4398a367d9ad6b3e5f71 in term 1.
I20260929 18:31:12.146750 2493 leader_election.cc:302] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd70bb0595a4398a367d9ad6b3e5f71, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:12.146924 2691 raft_consensus.cc:2781] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:12.147023 2691 raft_consensus.cc:686] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Running, Role: LEADER
I20260929 18:31:12.147161 2691 consensus_queue.cc:237] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.148762 2120 catalog_manager.cc:5495] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } health_report { overall_health: HEALTHY } } }
I20260929 18:31:12.170565 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:12.167315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.0.98.3:40267)
0929 18:31:12.167633 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:12.167639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:12.167652 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:12.167688 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:12.167690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:12.167729 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:12.167820 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:12.168071 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.168820 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.169688 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.169909 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.170002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:12.170208 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:12.170272 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:12.170330 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:12.170409 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:31:12.170453 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":169,"threads_started":1}
I20260929 18:31:12.179235 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:12.176051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51943 (local address 127.0.98.3:40267)
0929 18:31:12.176233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:12.176237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:12.176357 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:31:12.176479 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:12.176485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:12.176524 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:12.176611 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:12.176838 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.177335 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.178149 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.178378 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.178478 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:12.178736 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:12.178822 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:12.178898 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:12.178980 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:31:12.179072 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":81}
I20260929 18:31:12.179235 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:12.175943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40267 (local address 127.0.98.3:51943)
0929 18:31:12.176263 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:12.176276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:12.176398 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:12.176617 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:12.176619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:12.176625 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:12.176735 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:12.176741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.177443 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.177449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:12.178038 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:12.178045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.178489 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.178492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:12.178605 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:12.178627 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:12.178903 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:12.178911 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:31:12.178974 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:31:12.179080 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":43,"thread_start_us":112,"threads_started":1}
I20260929 18:31:12.184793 2279 raft_consensus.cc:1261] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Refusing update from remote peer 3dd70bb0595a4398a367d9ad6b3e5f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:12.184864 2419 raft_consensus.cc:1261] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Refusing update from remote peer 3dd70bb0595a4398a367d9ad6b3e5f71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:12.185250 2651 consensus_queue.cc:1035] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:12.185444 2691 consensus_queue.cc:1035] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:12.221710 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:12.221877 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:12.223798 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Sending DeleteTablet for 3 replicas of tablet 9a29c4f4eba4442e8eb13e89f466678a
I20260929 18:31:12.224323 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36958
I20260929 18:31:12.224462 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41956
I20260929 18:31:12.224512 2707 tablet_replica.cc:330] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: stopping tablet replica
I20260929 18:31:12.224619 2706 tablet_replica.cc:330] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: stopping tablet replica
I20260929 18:31:12.224640 2707 raft_consensus.cc:2229] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:12.224627 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29c4f4eba4442e8eb13e89f466678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45678
I20260929 18:31:12.224723 2706 raft_consensus.cc:2229] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:12.224749 2707 raft_consensus.cc:2258] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.224807 2705 tablet_replica.cc:330] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: stopping tablet replica
I20260929 18:31:12.224941 2705 raft_consensus.cc:2229] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:12.225000 2706 raft_consensus.cc:2258] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.225056 2705 raft_consensus.cc:2258] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:12.225502 2707 ts_tablet_manager.cc:1905] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.226150 2705 ts_tablet_manager.cc:1905] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.227555 2707 ts_tablet_manager.cc:1918] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:12.227648 2707 log.cc:1196] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/9a29c4f4eba4442e8eb13e89f466678a
I20260929 18:31:12.227892 2707 ts_tablet_manager.cc:1939] T 9a29c4f4eba4442e8eb13e89f466678a P f4cba27025db4a07828e7b6bf4ce7d7a: Deleting consensus metadata
I20260929 18:31:12.228597 2706 ts_tablet_manager.cc:1905] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:12.228639 2705 ts_tablet_manager.cc:1918] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:12.228706 2705 log.cc:1196] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/9a29c4f4eba4442e8eb13e89f466678a
I20260929 18:31:12.228684 2109 catalog_manager.cc:4841] TS f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085): tablet 9a29c4f4eba4442e8eb13e89f466678a (table test-table [id=4afd6a5ca5394e76b2ee569685f785d8]) successfully deleted
I20260929 18:31:12.228943 2705 ts_tablet_manager.cc:1939] T 9a29c4f4eba4442e8eb13e89f466678a P 408d1ecaf5df4ee7856944591f8de9a6: Deleting consensus metadata
I20260929 18:31:12.229631 2107 catalog_manager.cc:4841] TS 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699): tablet 9a29c4f4eba4442e8eb13e89f466678a (table test-table [id=4afd6a5ca5394e76b2ee569685f785d8]) successfully deleted
I20260929 18:31:12.230736 2706 ts_tablet_manager.cc:1918] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:12.230829 2706 log.cc:1196] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/9a29c4f4eba4442e8eb13e89f466678a
I20260929 18:31:12.231050 2706 ts_tablet_manager.cc:1939] T 9a29c4f4eba4442e8eb13e89f466678a P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting consensus metadata
I20260929 18:31:12.231658 2107 catalog_manager.cc:4841] TS 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267): tablet 9a29c4f4eba4442e8eb13e89f466678a (table test-table [id=4afd6a5ca5394e76b2ee569685f785d8]) successfully deleted
I20260929 18:31:12.231993 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:12.232467 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:12.236956 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 (DEFAULT_TABLE table=test-table [id=e6bc0ecd65a548379265e787062cec34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.236937 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 (DEFAULT_TABLE table=test-table [id=e6bc0ecd65a548379265e787062cec34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.237191 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dff58353afb40a9ac39d04ef7f53b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.237190 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dff58353afb40a9ac39d04ef7f53b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.237129 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 (DEFAULT_TABLE table=test-table [id=e6bc0ecd65a548379265e787062cec34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:12.237398 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dff58353afb40a9ac39d04ef7f53b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:12.239342 2716 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Bootstrap starting.
I20260929 18:31:12.239748 2717 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Bootstrap starting.
I20260929 18:31:12.240296 2716 tablet_bootstrap.cc:654] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.240839 2717 tablet_bootstrap.cc:654] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.240862 2718 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: Bootstrap starting.
I20260929 18:31:12.241525 2716 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: No bootstrap required, opened a new log
I20260929 18:31:12.241612 2716 ts_tablet_manager.cc:1397] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:12.241753 2718 tablet_bootstrap.cc:654] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:12.241886 2717 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: No bootstrap required, opened a new log
I20260929 18:31:12.241956 2717 ts_tablet_manager.cc:1397] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:12.242040 2716 raft_consensus.cc:348] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.242136 2716 raft_consensus.cc:374] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.242168 2716 raft_consensus.cc:729] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1ecaf5df4ee7856944591f8de9a6, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.242290 2716 consensus_queue.cc:260] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.242312 2717 raft_consensus.cc:348] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.242398 2717 raft_consensus.cc:374] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.242440 2717 raft_consensus.cc:729] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd70bb0595a4398a367d9ad6b3e5f71, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.242558 2716 ts_tablet_manager.cc:1428] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:12.242549 2717 consensus_queue.cc:260] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.242785 2717 ts_tablet_manager.cc:1428] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:12.244035 2718 tablet_bootstrap.cc:492] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: No bootstrap required, opened a new log
I20260929 18:31:12.244222 2718 ts_tablet_manager.cc:1397] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:12.244689 2718 raft_consensus.cc:348] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.244856 2718 raft_consensus.cc:374] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:12.244933 2718 raft_consensus.cc:729] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Initialized, Role: FOLLOWER
I20260929 18:31:12.245072 2718 consensus_queue.cc:260] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.245347 2718 ts_tablet_manager.cc:1428] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:31:12.544289 2696 raft_consensus.cc:482] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:12.544489 2696 raft_consensus.cc:504] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.544945 2696 leader_election.cc:288] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699), 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267)
I20260929 18:31:12.545493 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dff58353afb40a9ac39d04ef7f53b23" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6" is_pre_election: true
I20260929 18:31:12.545492 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dff58353afb40a9ac39d04ef7f53b23" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" is_pre_election: true
I20260929 18:31:12.545677 2419 raft_consensus.cc:2454] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:12.545693 2559 raft_consensus.cc:2454] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 0.
I20260929 18:31:12.546001 2211 leader_election.cc:302] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408d1ecaf5df4ee7856944591f8de9a6, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:12.546214 2696 raft_consensus.cc:2781] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:12.546289 2696 raft_consensus.cc:482] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:12.546339 2696 raft_consensus.cc:3037] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.547364 2696 raft_consensus.cc:504] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.547680 2696 leader_election.cc:288] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Requested vote from peers 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699), 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267)
I20260929 18:31:12.548036 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dff58353afb40a9ac39d04ef7f53b23" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1ecaf5df4ee7856944591f8de9a6"
I20260929 18:31:12.548089 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dff58353afb40a9ac39d04ef7f53b23" candidate_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71"
I20260929 18:31:12.548195 2419 raft_consensus.cc:3037] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.548243 2559 raft_consensus.cc:3037] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:12.549063 2419 raft_consensus.cc:2454] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:12.549185 2559 raft_consensus.cc:2454] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4cba27025db4a07828e7b6bf4ce7d7a in term 1.
I20260929 18:31:12.549355 2211 leader_election.cc:302] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408d1ecaf5df4ee7856944591f8de9a6, f4cba27025db4a07828e7b6bf4ce7d7a; no voters:
I20260929 18:31:12.549518 2696 raft_consensus.cc:2781] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:12.549603 2696 raft_consensus.cc:686] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Becoming Leader. State: Replica: f4cba27025db4a07828e7b6bf4ce7d7a, State: Running, Role: LEADER
I20260929 18:31:12.549737 2696 consensus_queue.cc:237] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } }
I20260929 18:31:12.551468 2122 catalog_manager.cc:5495] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cba27025db4a07828e7b6bf4ce7d7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:12.615224 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:12.612020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.0.98.1:46085)
0929 18:31:12.612329 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:12.612333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:12.612345 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:31:12.612391 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:12.612393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:12.612431 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:12.612520 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:12.612761 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.613274 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.614162 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:12.614415 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:12.614524 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:12.614764 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:12.614846 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:12.614924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:12.615012 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:31:12.615063 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":128,"threads_started":1}
I20260929 18:31:12.620854 2559 raft_consensus.cc:1261] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:12.620878 2419 raft_consensus.cc:1261] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Refusing update from remote peer f4cba27025db4a07828e7b6bf4ce7d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:12.621234 2699 consensus_queue.cc:1035] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd70bb0595a4398a367d9ad6b3e5f71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:12.621394 2696 consensus_queue.cc:1035] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1ecaf5df4ee7856944591f8de9a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:31:12.633138 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:31:14.180097 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:14.180264 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:14.182638 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6218658cc659415096f8b1a880ae7f03: Sending DeleteTablet for 3 replicas of tablet 8dff58353afb40a9ac39d04ef7f53b23
I20260929 18:31:14.183125 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36958
I20260929 18:31:14.183296 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41956
I20260929 18:31:14.183461 2762 tablet_replica.cc:330] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a: stopping tablet replica
I20260929 18:31:14.183446 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dff58353afb40a9ac39d04ef7f53b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45678
I20260929 18:31:14.183579 2762 raft_consensus.cc:2229] T 8dff58353afb40a9ac39d04ef7f53b23 P f4cba27025db4a07828e7b6bf4ce7d7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:14.183630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2194
I20260929 18:31:14.186319 2764 tablet_replica.cc:330] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: stopping tablet replica
I20260929 18:31:14.186465 2764 raft_consensus.cc:2229] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:14.186595 2764 raft_consensus.cc:2258] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:14.186859 2763 tablet_replica.cc:330] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: stopping tablet replica
I20260929 18:31:14.187026 2763 raft_consensus.cc:2229] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:14.187152 2763 raft_consensus.cc:2258] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:14.187474 2764 ts_tablet_manager.cc:1905] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:14.187711 2763 ts_tablet_manager.cc:1905] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:14.190120 2764 ts_tablet_manager.cc:1918] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:31:14.190204 2764 log.cc:1196] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/8dff58353afb40a9ac39d04ef7f53b23
I20260929 18:31:14.190201 2763 ts_tablet_manager.cc:1918] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:31:14.190263 2763 log.cc:1196] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/8dff58353afb40a9ac39d04ef7f53b23
I20260929 18:31:14.190541 2763 ts_tablet_manager.cc:1939] T 8dff58353afb40a9ac39d04ef7f53b23 P 3dd70bb0595a4398a367d9ad6b3e5f71: Deleting consensus metadata
I20260929 18:31:14.190541 2764 ts_tablet_manager.cc:1939] T 8dff58353afb40a9ac39d04ef7f53b23 P 408d1ecaf5df4ee7856944591f8de9a6: Deleting consensus metadata
I20260929 18:31:14.191222 2107 catalog_manager.cc:4841] TS 3dd70bb0595a4398a367d9ad6b3e5f71 (127.0.98.3:40267): tablet 8dff58353afb40a9ac39d04ef7f53b23 (table test-table [id=e6bc0ecd65a548379265e787062cec34]) successfully deleted
W20260929 18:31:14.191252 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46085 (error 108)
I20260929 18:31:14.191423 2107 catalog_manager.cc:4841] TS 408d1ecaf5df4ee7856944591f8de9a6 (127.0.98.2:45699): tablet 8dff58353afb40a9ac39d04ef7f53b23 (table test-table [id=e6bc0ecd65a548379265e787062cec34]) successfully deleted
I20260929 18:31:14.191562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2334
I20260929 18:31:14.197120 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:14.196696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46085 (local address 127.0.0.1:36972)
0929 18:31:14.196966 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:14.197033 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
W20260929 18:31:14.197405 2109 catalog_manager.cc:4584] TS f4cba27025db4a07828e7b6bf4ce7d7a (127.0.98.1:46085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8dff58353afb40a9ac39d04ef7f53b23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46085: connect: Connection refused (error 111)
I20260929 18:31:14.197674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2474
I20260929 18:31:14.204527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2090
2026-09-29T18:31:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6661 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:14 dist-test-slave-fmv8 krb5kdc[2769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:14 dist-test-slave-fmv8 krb5kdc[2769](info): set up 2 sockets
Sep 29 18:31:14 dist-test-slave-fmv8 krb5kdc[2769](info): commencing operation
krb5kdc: starting...
W20260929 18:31:15.236017 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:15.366474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:15.471967 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:15.472337 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:15.472425 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:15.476470 2782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:15.476565 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:15.476603 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:15.476640 2782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:15.476672 2782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:15.481885 2782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:15.482929 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:15.483939 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:31:15.492759 2782 server_base.cc:1034] running on GCE node
W20260929 18:31:15.492710 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:15.492717 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:15.492728 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:15.493554 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:15.494649 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:15.495858 2782 hybrid_clock.cc:648] HybridClock initialized: now 1790706675495828 us; error 43 us; skew 500 ppm
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:15.498664 2782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:15.499601 2782 webserver.cc:460] Webserver started at http://127.0.98.62:34665/ using document root <none> and password file <none>
I20260929 18:31:15.500247 2782 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:15.500341 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:15.500571 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:15.502593 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "b40ad0365c72428499e3e0ed7de2e3e4"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "7fb3d066f1bf54ae5887f9e92028efc2"
server_key_iv: "e54b48466b074b5dfa74a44f25178cba"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.503185 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "b40ad0365c72428499e3e0ed7de2e3e4"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "7fb3d066f1bf54ae5887f9e92028efc2"
server_key_iv: "e54b48466b074b5dfa74a44f25178cba"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.506893 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:15.509331 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.510468 2782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:15.510623 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "b40ad0365c72428499e3e0ed7de2e3e4"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "7fb3d066f1bf54ae5887f9e92028efc2"
server_key_iv: "e54b48466b074b5dfa74a44f25178cba"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.510735 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:15.520589 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:15.523443 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:15.523725 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:15.530889 2782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45217
I20260929 18:31:15.530907 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45217 every 8 connection(s)
I20260929 18:31:15.531845 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:15.534520 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:15.539721 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: Bootstrap starting.
I20260929 18:31:15.541728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2782
I20260929 18:31:15.541857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:15.542114 392 external_mini_cluster.cc:1428] Setting key 5599fa4cdb957e8472add3c30a02c5e8
I20260929 18:31:15.542140 2850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:15.542901 2850 log.cc:824] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:15.544847 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: No bootstrap required, opened a new log
I20260929 18:31:15.548127 2850 raft_consensus.cc:348] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } }
I20260929 18:31:15.548420 2850 raft_consensus.cc:374] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:15.548502 2850 raft_consensus.cc:729] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40ad0365c72428499e3e0ed7de2e3e4, State: Initialized, Role: FOLLOWER
I20260929 18:31:15.548976 2850 consensus_queue.cc:260] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } }
I20260929 18:31:15.549127 2850 raft_consensus.cc:388] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:15.549213 2850 raft_consensus.cc:482] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:15.549309 2850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:15.550148 2850 raft_consensus.cc:504] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } }
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:15.550513 2850 leader_election.cc:302] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b40ad0365c72428499e3e0ed7de2e3e4; no voters:
I20260929 18:31:15.550858 2850 leader_election.cc:288] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:15.550963 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:15.551177 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [term 1 LEADER]: Becoming Leader. State: Replica: b40ad0365c72428499e3e0ed7de2e3e4, State: Running, Role: LEADER
I20260929 18:31:15.551548 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } }
I20260929 18:31:15.552136 2850 sys_catalog.cc:564] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:15.553923 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } } }
I20260929 18:31:15.554049 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:15.554634 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b40ad0365c72428499e3e0ed7de2e3e4. Latest consensus state: current_term: 1 leader_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b40ad0365c72428499e3e0ed7de2e3e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45217 } } }
I20260929 18:31:15.555267 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:15.556912 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:15.543257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.98.62:45217)
0929 18:31:15.543689 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:15.543705 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:15.543738 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:31:15.544409 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:15.544427 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:15.544769 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:15.545122 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:15.545886 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.546808 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.547669 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.548129 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.548350 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:15.551066 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:15.551072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:15.551090 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:15.551121 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:15.553319 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:15.553876 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:15.553884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:15.553890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:15.554030 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:15.554356 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:15.554359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:15.554361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:15.554885 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:15.555049 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:15.555708 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 18:31:15.555989 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
W20260929 18:31:15.557168 2869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:15.557235 2869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:15.557341 2870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:15.558212 2870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:15.563453 2870 catalog_manager.cc:1347] Generated new cluster ID: 109cc658d52042c0ab423258d508c70f
I20260929 18:31:15.563515 2870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:15.580184 2870 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:15.581113 2870 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:15.588866 2870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b40ad0365c72428499e3e0ed7de2e3e4: Generated new TSK 0
I20260929 18:31:15.589610 2870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:15.614930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:15.723791 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:15.724037 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:15.724078 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:15.728256 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:15.728309 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:15.728394 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:15.733665 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:15.734694 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:15.735762 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:15.745239 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:15.745352 2876 server_base.cc:1034] running on GCE node
W20260929 18:31:15.745270 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:15.745265 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:15.745900 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:15.746506 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:15.747668 2876 hybrid_clock.cc:648] HybridClock initialized: now 1790706675747639 us; error 44 us; skew 500 ppm
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:15.750527 2876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:15.751435 2876 webserver.cc:460] Webserver started at http://127.0.98.1:44931/ using document root <none> and password file <none>
I20260929 18:31:15.752009 2876 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:15.752122 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:15.752434 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:15.754572 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fa1c955b3584e4daaa7e48ba80e365d"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "31a28995ad6d3fe8390bec19c959cf9e"
server_key_iv: "11921a194cc755b5f5413cd8ecf05262"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.755142 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fa1c955b3584e4daaa7e48ba80e365d"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "31a28995ad6d3fe8390bec19c959cf9e"
server_key_iv: "11921a194cc755b5f5413cd8ecf05262"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.759159 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:31:15.761574 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.763065 2876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:31:15.763238 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "4fa1c955b3584e4daaa7e48ba80e365d"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "31a28995ad6d3fe8390bec19c959cf9e"
server_key_iv: "11921a194cc755b5f5413cd8ecf05262"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.763378 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:15.778674 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:15.781859 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:15.782055 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:15.782622 2876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:15.783411 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:15.783479 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.783546 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:15.783601 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.792429 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45221
I20260929 18:31:15.792471 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45221 every 8 connection(s)
I20260929 18:31:15.793265 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:15.800941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 2876
I20260929 18:31:15.801033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:15.801316 392 external_mini_cluster.cc:1428] Setting key 1b88a3bf874715c21321c633e373e5b4
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:15.807404 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:15.794943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46307 (local address 127.0.98.62:45217)
0929 18:31:15.795216 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:15.795221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:15.796024 (+ 803us) server_negotiation.cc:407] Connection header received
0929 18:31:15.797478 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:15.797482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:15.797538 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:15.797644 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:15.798777 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.799303 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.800448 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.800681 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.800810 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:15.803479 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:15.803485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:15.803492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:15.803524 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:15.805107 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:15.805750 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:15.805756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:15.805761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:15.805858 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:15.806180 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:15.806186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:15.806191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:15.806448 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:15.806563 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:15.807095 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 18:31:15.807226 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260929 18:31:15.808073 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:15.795431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45217 (local address 127.0.98.1:46307)
0929 18:31:15.795861 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:15.795920 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:15.797271 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:15.797817 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:15.797825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:15.798179 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:15.798628 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:15.798640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.799428 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.799433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:15.800321 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:15.800330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:15.800845 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:15.800849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:15.801051 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:15.801675 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:15.801691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:15.803317 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:15.805234 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:15.805241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:15.805257 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:15.805630 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:15.805953 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:15.805955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:15.805957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:15.806084 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:15.806553 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:15.806557 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:15.806811 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 18:31:15.807001 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:31:15.807246 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260929 18:31:15.809321 3005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45217
I20260929 18:31:15.809588 3005 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:15.810148 3005 heartbeater.cc:511] Master 127.0.98.62:45217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:15.811743 2814 ts_manager.cc:194] Registered new tserver with Master: 4fa1c955b3584e4daaa7e48ba80e365d (127.0.98.1:45221)
I20260929 18:31:15.812949 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:15.827811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:15.936412 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:15.936709 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:15.936785 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:15.940801 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:15.940888 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:15.940975 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:15.946295 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:15.947290 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:15.948349 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:15.958142 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:15.958163 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:15.958251 3010 server_base.cc:1034] running on GCE node
W20260929 18:31:15.958199 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:15.958725 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:15.959326 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:15.960489 3010 hybrid_clock.cc:648] HybridClock initialized: now 1790706675960451 us; error 50 us; skew 500 ppm
Sep 29 18:31:15 dist-test-slave-fmv8 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:15.963380 3010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:15.964298 3010 webserver.cc:460] Webserver started at http://127.0.98.2:33433/ using document root <none> and password file <none>
I20260929 18:31:15.964857 3010 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:15.964917 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:15.965130 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:15.967191 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "abd79946a85146f794ce89580be3692a"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "723d9ad492f7f4ecbbfa347d147fa230"
server_key_iv: "b45049743b256eebc78c728663f1c969"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.967784 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "abd79946a85146f794ce89580be3692a"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "723d9ad492f7f4ecbbfa347d147fa230"
server_key_iv: "b45049743b256eebc78c728663f1c969"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.971652 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:31:15.974320 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.975561 3010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:15.975668 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "abd79946a85146f794ce89580be3692a"
format_stamp: "Formatted at 2026-09-29 18:31:15 on dist-test-slave-fmv8"
server_key: "723d9ad492f7f4ecbbfa347d147fa230"
server_key_iv: "b45049743b256eebc78c728663f1c969"
server_key_version: "encryptionkey@0"
I20260929 18:31:15.975819 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:15.985637 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:15.988922 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:15.989135 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:15.989722 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:15.990514 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:15.990566 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:15.990613 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:15.990667 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:16.000653 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46781
I20260929 18:31:16.000742 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46781 every 8 connection(s)
I20260929 18:31:16.001614 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:16.003609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3010
I20260929 18:31:16.003736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:16.004035 392 external_mini_cluster.cc:1428] Setting key 5817b0feb8dddec691d01e573e55881a
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:31:16.017942 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:16.003684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56857 (local address 127.0.98.62:45217)
0929 18:31:16.003848 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:16.003853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:16.004952 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 18:31:16.006344 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:16.006350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:16.006409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:16.006500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:16.008278 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.008856 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.010732 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.010940 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.011022 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:16.014030 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:16.014040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:16.014043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:16.014067 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:16.015796 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:16.016404 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:16.016407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:16.016409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:16.016492 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:16.016859 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:16.016862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:16.016864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:16.017044 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:16.017160 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:16.017656 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 18:31:16.017778 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:31:16.018883 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:16.004238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45217 (local address 127.0.98.2:56857)
0929 18:31:16.004796 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:16.004850 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:16.006127 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:16.006700 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:16.006707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:16.007233 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:16.007996 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:16.008019 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.008971 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.008977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:16.010591 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:16.010605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.011105 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.011111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:16.011354 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:16.012070 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:16.012094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:16.013855 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:16.015936 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:16.015942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:16.015957 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:16.016299 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:16.016587 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:16.016591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:16.016600 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:16.016774 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:16.017156 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:16.017163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:16.017422 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 18:31:16.017611 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:31:16.017938 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":174,"threads_started":1}
I20260929 18:31:16.020087 3139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45217
I20260929 18:31:16.020439 3139 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:16.020927 3139 heartbeater.cc:511] Master 127.0.98.62:45217 requested a full tablet report, sending...
I20260929 18:31:16.022043 2814 ts_manager.cc:194] Registered new tserver with Master: abd79946a85146f794ce89580be3692a (127.0.98.2:46781)
I20260929 18:31:16.022852 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:16.031796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:16.141374 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:16.141614 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:16.141655 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:16.145721 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:16.145779 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:16.145835 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:16.150908 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:16.151835 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:16.152992 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:16.162209 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:16.162220 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:16.162214 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:16.162403 3144 server_base.cc:1034] running on GCE node
I20260929 18:31:16.162954 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:16.163580 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:16.164788 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790706676164774 us; error 28 us; skew 500 ppm
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:16.167562 3144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:16.168579 3144 webserver.cc:460] Webserver started at http://127.0.98.3:33341/ using document root <none> and password file <none>
I20260929 18:31:16.169179 3144 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:16.169271 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:16.169476 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:16.171387 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f0dd27088dd427c88653f76d00b2995"
format_stamp: "Formatted at 2026-09-29 18:31:16 on dist-test-slave-fmv8"
server_key: "79141ca219157ca632c535273de3e669"
server_key_iv: "4d207696118ceba39945d1c632de18bb"
server_key_version: "encryptionkey@0"
I20260929 18:31:16.172052 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f0dd27088dd427c88653f76d00b2995"
format_stamp: "Formatted at 2026-09-29 18:31:16 on dist-test-slave-fmv8"
server_key: "79141ca219157ca632c535273de3e669"
server_key_iv: "4d207696118ceba39945d1c632de18bb"
server_key_version: "encryptionkey@0"
I20260929 18:31:16.175563 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:16.177980 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:16.179220 3144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:31:16.179349 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "5f0dd27088dd427c88653f76d00b2995"
format_stamp: "Formatted at 2026-09-29 18:31:16 on dist-test-slave-fmv8"
server_key: "79141ca219157ca632c535273de3e669"
server_key_iv: "4d207696118ceba39945d1c632de18bb"
server_key_version: "encryptionkey@0"
I20260929 18:31:16.179454 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:16.192127 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:16.195016 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:16.195209 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:16.195755 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:16.196591 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:16.196659 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:16.196722 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:16.196774 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:16.205740 3144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42133
I20260929 18:31:16.205761 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42133 every 8 connection(s)
I20260929 18:31:16.206578 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:16.207666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3144
I20260929 18:31:16.207779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:16.208103 392 external_mini_cluster.cc:1428] Setting key 533e3688333f568c18ef1f0d17c9cc43
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:16.220082 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:16.208452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46105 (local address 127.0.98.62:45217)
0929 18:31:16.208621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:16.208626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:16.209664 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 18:31:16.211233 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:16.211236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:16.211278 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:16.211347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:16.212450 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.212954 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.214021 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.214265 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.214363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:16.216862 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:16.216865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:16.216867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:16.216888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:16.218214 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:16.218655 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:16.218658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:16.218659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:16.218739 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:16.219027 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:16.219029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:16.219030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:16.219305 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:16.219384 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:16.219834 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:31:16.219942 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:31:16.220798 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:16.208932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45217 (local address 127.0.98.3:46105)
0929 18:31:16.209487 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:16.209560 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:16.211017 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:16.211475 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:16.211481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:16.211826 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:16.212324 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:16.212338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.213056 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.213059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:16.213923 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:16.213932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:16.214425 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:16.214429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:16.214572 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:16.215220 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:16.215236 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:16.216699 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:16.218310 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:16.218315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:16.218325 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:16.218563 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:16.218825 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:16.218827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:16.218828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:16.218949 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:16.219393 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:16.219398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:16.219604 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 18:31:16.219751 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 18:31:16.219980 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":134,"threads_started":1}
I20260929 18:31:16.221915 3273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45217
I20260929 18:31:16.222162 3273 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:16.222613 3273 heartbeater.cc:511] Master 127.0.98.62:45217 requested a full tablet report, sending...
I20260929 18:31:16.223528 2814 ts_manager.cc:194] Registered new tserver with Master: 5f0dd27088dd427c88653f76d00b2995 (127.0.98.3:42133)
I20260929 18:31:16.224057 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46105
I20260929 18:31:16.233883 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:31:16.246613 2854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:16.245310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.98.62:45217)
0929 18:31:16.245454 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:16.245458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:16.245518 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:31:16.245691 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:16.245694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:16.245751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:16.245834 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:16.246502 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38444: BlockingRecv error: recv got EOF from 127.0.0.1:38444 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:31:16.247766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2876
I20260929 18:31:16.253461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3010
I20260929 18:31:16.258872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3144
I20260929 18:31:16.264026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 2782
2026-09-29T18:31:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2058 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[3285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[3285](info): set up 2 sockets
Sep 29 18:31:16 dist-test-slave-fmv8 krb5kdc[3285](info): commencing operation
krb5kdc: starting...
W20260929 18:31:17.304219 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:17.420568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:17.526062 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:17.526342 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:17.526418 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:17.530452 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:17.530536 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:17.530573 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:17.530632 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:17.530671 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:17.535787 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:17.536824 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:17.537789 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:17.545917 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:17.546097 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:17.546195 3298 server_base.cc:1034] running on GCE node
W20260929 18:31:17.545922 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:17.546803 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:17.547894 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:17.549062 3298 hybrid_clock.cc:648] HybridClock initialized: now 1790706677549042 us; error 34 us; skew 500 ppm
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:17.555943 3298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:17.556921 3298 webserver.cc:460] Webserver started at http://127.0.98.62:44891/ using document root <none> and password file <none>
I20260929 18:31:17.557475 3298 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:17.557538 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:17.557746 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:17.559759 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "60d064903ba04d62a96277d4755733b3"
format_stamp: "Formatted at 2026-09-29 18:31:17 on dist-test-slave-fmv8"
server_key: "eae94762ff40d9708df56235a433091d"
server_key_iv: "4814742fe08f6bf06a87a1576ecbd59d"
server_key_version: "encryptionkey@0"
I20260929 18:31:17.560349 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "60d064903ba04d62a96277d4755733b3"
format_stamp: "Formatted at 2026-09-29 18:31:17 on dist-test-slave-fmv8"
server_key: "eae94762ff40d9708df56235a433091d"
server_key_iv: "4814742fe08f6bf06a87a1576ecbd59d"
server_key_version: "encryptionkey@0"
I20260929 18:31:17.563834 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:17.566288 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:17.567495 3298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:17.567639 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "60d064903ba04d62a96277d4755733b3"
format_stamp: "Formatted at 2026-09-29 18:31:17 on dist-test-slave-fmv8"
server_key: "eae94762ff40d9708df56235a433091d"
server_key_iv: "4814742fe08f6bf06a87a1576ecbd59d"
server_key_version: "encryptionkey@0"
I20260929 18:31:17.567746 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:17.580720 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:17.583808 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:17.584004 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:17.591239 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41819
I20260929 18:31:17.591565 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41819 every 8 connection(s)
I20260929 18:31:17.592247 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:17.594856 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:17.596213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3298
I20260929 18:31:17.596357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:17.596629 392 external_mini_cluster.cc:1428] Setting key c0c36d48d56af35aa7df481f8e192337
I20260929 18:31:17.601133 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3: Bootstrap starting.
I20260929 18:31:17.603729 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:17.604498 3366 log.cc:824] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:17.606245 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3: No bootstrap required, opened a new log
I20260929 18:31:17.610961 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:17.597861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.0.98.62:41819)
0929 18:31:17.598257 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:17.598275 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:17.598316 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:31:17.598921 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:17.598937 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:17.599242 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:17.599514 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:17.600428 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:17.601568 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:17.602719 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:17.603173 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:17.603296 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:17.605873 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:17.605878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:17.605891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:17.605915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:17.607899 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:17.608422 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:17.608430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:17.608438 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:17.608601 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:17.608890 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:17.608894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:17.608896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:17.609370 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:17.609568 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:17.609778 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 18:31:17.610058 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":144,"threads_started":1}
I20260929 18:31:17.610880 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } }
I20260929 18:31:17.611131 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:17.611210 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d064903ba04d62a96277d4755733b3, State: Initialized, Role: FOLLOWER
I20260929 18:31:17.611645 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } }
I20260929 18:31:17.611788 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:17.611869 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:17.611971 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:17.612834 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } }
I20260929 18:31:17.613220 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d064903ba04d62a96277d4755733b3; no voters:
I20260929 18:31:17.613559 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:17.613623 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:17.613826 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [term 1 LEADER]: Becoming Leader. State: Replica: 60d064903ba04d62a96277d4755733b3, State: Running, Role: LEADER
I20260929 18:31:17.613885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:31:17.614152 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } }
I20260929 18:31:17.614765 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:17.615777 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d064903ba04d62a96277d4755733b3. Latest consensus state: current_term: 1 leader_uuid: "60d064903ba04d62a96277d4755733b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } } }
I20260929 18:31:17.615911 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:17.616237 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d064903ba04d62a96277d4755733b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d064903ba04d62a96277d4755733b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41819 } } }
I20260929 18:31:17.616321 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:17.616335 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:17.619406 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:17.625383 3381 catalog_manager.cc:1347] Generated new cluster ID: bb6b9ec9b2fd4fd1932e04ce818f68ff
I20260929 18:31:17.625444 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:17.637025 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:17.631657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.98.62:41819)
0929 18:31:17.631766 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:17.631770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:17.631959 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:31:17.632066 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:17.632069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:17.632118 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:17.632234 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:17.632751 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:17.633543 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:17.634646 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:17.635018 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:17.635121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:17.635988 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:17.635992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:17.635995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:17.636571 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:17.636605 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:17.636624 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:17.636714 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:17.636825 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:31:17.636880 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:31:17.639173 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:17.640017 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:17.648772 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d064903ba04d62a96277d4755733b3: Generated new TSK 0
I20260929 18:31:17.649390 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:17.659456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3298
2026-09-29T18:31:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1393 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3399](info): set up 2 sockets
Sep 29 18:31:17 dist-test-slave-fmv8 krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260929 18:31:18.696053 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:18 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:18.812986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:18.920061 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:18.920432 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:18.920504 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:18.925266 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:18.925324 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:18.925344 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:18.925361 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:18.925376 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:18.930567 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:18.931478 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:18.932493 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:18.940660 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:18.940642 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:18.940871 3412 server_base.cc:1034] running on GCE node
W20260929 18:31:18.940634 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:18.941558 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:18.942605 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:18.943805 3412 hybrid_clock.cc:648] HybridClock initialized: now 1790706678943760 us; error 57 us; skew 500 ppm
Sep 29 18:31:18 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:18.946513 3412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:18.947336 3412 webserver.cc:460] Webserver started at http://127.0.98.62:43565/ using document root <none> and password file <none>
I20260929 18:31:18.947868 3412 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:18.947956 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:18.948231 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:18.950204 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49"
format_stamp: "Formatted at 2026-09-29 18:31:18 on dist-test-slave-fmv8"
server_key: "12b4b107c15166fb282124a7f023ea76"
server_key_iv: "07881ae56b2032bda3152c39a44de1a6"
server_key_version: "encryptionkey@0"
I20260929 18:31:18.950853 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49"
format_stamp: "Formatted at 2026-09-29 18:31:18 on dist-test-slave-fmv8"
server_key: "12b4b107c15166fb282124a7f023ea76"
server_key_iv: "07881ae56b2032bda3152c39a44de1a6"
server_key_version: "encryptionkey@0"
I20260929 18:31:18.954475 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 18:31:18.957175 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:18.958308 3412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:18.958427 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49"
format_stamp: "Formatted at 2026-09-29 18:31:18 on dist-test-slave-fmv8"
server_key: "12b4b107c15166fb282124a7f023ea76"
server_key_iv: "07881ae56b2032bda3152c39a44de1a6"
server_key_version: "encryptionkey@0"
I20260929 18:31:18.958525 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:18.980357 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:18.983793 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:18.984005 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:18.991106 3412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33197
I20260929 18:31:18.991400 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33197 every 8 connection(s)
I20260929 18:31:18.992111 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:18.995045 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:18.999012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3412
I20260929 18:31:18.999140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:18.999457 392 external_mini_cluster.cc:1428] Setting key 389e9b2deb7b4cd1020b0e8dda09c05c
I20260929 18:31:19.001560 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49: Bootstrap starting.
I20260929 18:31:19.003818 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:19.004633 3480 log.cc:824] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:19.006788 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49: No bootstrap required, opened a new log
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:19.010830 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } }
I20260929 18:31:19.011085 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:19.011166 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1340cd3cf7da41bf9949ce2b3bc4bc49, State: Initialized, Role: FOLLOWER
I20260929 18:31:19.011660 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } }
I20260929 18:31:19.011826 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:19.011910 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:19.012006 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:19.012934 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } }
I20260929 18:31:19.013281 3480 leader_election.cc:302] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1340cd3cf7da41bf9949ce2b3bc4bc49; no voters:
I20260929 18:31:19.013634 3480 leader_election.cc:288] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:19.014019 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:19.014219 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [term 1 LEADER]: Becoming Leader. State: Replica: 1340cd3cf7da41bf9949ce2b3bc4bc49, State: Running, Role: LEADER
I20260929 18:31:19.014568 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } }
I20260929 18:31:19.014833 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.001027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.0.98.62:33197)
0929 18:31:19.001480 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:19.001497 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:19.001533 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:31:19.002239 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:19.002256 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:19.002606 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:19.002980 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:19.003790 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.004959 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.006176 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.006630 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.006747 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.009630 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:19.009635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:19.009647 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:19.009674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:19.011542 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.012063 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.012068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.012073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.012253 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.012536 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.012540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.012542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.012974 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:19.013189 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:19.013471 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:31:19.013797 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260929 18:31:19.014954 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:19.015745 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1340cd3cf7da41bf9949ce2b3bc4bc49. Latest consensus state: current_term: 1 leader_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } } }
I20260929 18:31:19.015926 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:19.016225 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1340cd3cf7da41bf9949ce2b3bc4bc49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33197 } } }
I20260929 18:31:19.016333 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:19.017201 3494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:19.020179 3494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:19.025273 3494 catalog_manager.cc:1347] Generated new cluster ID: bd997ab7971a4406ac16f6d6cc78b8cd
I20260929 18:31:19.025338 3494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:19.055933 3494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:19.056967 3494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:19.061292 3494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1340cd3cf7da41bf9949ce2b3bc4bc49: Generated new TSK 0
I20260929 18:31:19.062039 3494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:19.091224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:19.208050 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:19.208336 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:19.208379 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:19.212425 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:19.212478 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:19.212559 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:19.217948 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:19.218919 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:19.219902 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:19.229331 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:19.229375 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:19.229411 3506 server_base.cc:1034] running on GCE node
W20260929 18:31:19.229336 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:19.229929 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:19.230547 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:19.231755 3506 hybrid_clock.cc:648] HybridClock initialized: now 1790706679231734 us; error 31 us; skew 500 ppm
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:19.234483 3506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:19.235342 3506 webserver.cc:460] Webserver started at http://127.0.98.1:46499/ using document root <none> and password file <none>
I20260929 18:31:19.236001 3506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:19.236085 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:19.236343 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:19.238231 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "d139c7ee51c7426f8adcfed09f7a6fc6"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "e9d668c295008813b45264690e0be52e"
server_key_iv: "9cdc1784087c0b434138669ee1b76e35"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.238720 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d139c7ee51c7426f8adcfed09f7a6fc6"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "e9d668c295008813b45264690e0be52e"
server_key_iv: "9cdc1784087c0b434138669ee1b76e35"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.242277 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:19.244660 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.245852 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:19.245967 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "d139c7ee51c7426f8adcfed09f7a6fc6"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "e9d668c295008813b45264690e0be52e"
server_key_iv: "9cdc1784087c0b434138669ee1b76e35"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.246071 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:19.267037 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:19.269948 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:19.270146 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:19.270761 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:19.271569 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:19.271616 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.271656 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:19.271670 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.281069 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34753
I20260929 18:31:19.281098 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34753 every 8 connection(s)
I20260929 18:31:19.281917 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:19.287911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3506
I20260929 18:31:19.288055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:19.288400 392 external_mini_cluster.cc:1428] Setting key c3fc42e8bf2aa2399e784e432421cf04
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:19.295570 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.283655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41415 (local address 127.0.98.62:33197)
0929 18:31:19.283785 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:19.283788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:19.284578 (+ 790us) server_negotiation.cc:407] Connection header received
0929 18:31:19.285984 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:19.285987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:19.286042 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:19.286154 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:19.287344 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.287870 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.289176 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.289410 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.289524 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.291964 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:19.291967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:19.291969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:19.291993 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:19.293532 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.294043 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.294048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.294050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.294133 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.294495 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.294499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.294501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.294680 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:19.294797 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:19.295282 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:31:19.295402 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:19.296417 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.283942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33197 (local address 127.0.98.1:41415)
0929 18:31:19.284426 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:19.284469 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:19.285774 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:19.286325 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:19.286332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:19.286682 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:19.287181 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.287195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.288011 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.288014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.289040 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.289049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.289557 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.289561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.289728 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.290313 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:19.290330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:19.291790 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:19.293654 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.293659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.293673 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.293934 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.294258 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.294261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.294263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.294398 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.294794 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:19.294800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:19.295034 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 18:31:19.295239 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 18:31:19.295548 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:19.297713 3635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33197
I20260929 18:31:19.297982 3635 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:19.298519 3635 heartbeater.cc:511] Master 127.0.98.62:33197 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:31:19.300256 3444 ts_manager.cc:194] Registered new tserver with Master: d139c7ee51c7426f8adcfed09f7a6fc6 (127.0.98.1:34753)
I20260929 18:31:19.301429 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:19.314533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:19.424499 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:19.424751 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:19.424791 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:19.428736 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:19.428785 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:19.428871 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:19.433872 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:19.434823 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:19.435783 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:19.445232 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:19.445232 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:19.445228 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:19.445501 3640 server_base.cc:1034] running on GCE node
I20260929 18:31:19.445960 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:19.446626 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:19.447818 3640 hybrid_clock.cc:648] HybridClock initialized: now 1790706679447788 us; error 39 us; skew 500 ppm
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:19.450613 3640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:19.451467 3640 webserver.cc:460] Webserver started at http://127.0.98.2:46197/ using document root <none> and password file <none>
I20260929 18:31:19.452024 3640 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:19.452123 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:19.452382 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:19.454286 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "bdafe788482b4fd8a04f287f3d289883"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "944ddaac54499732be9580bd3915790e"
server_key_iv: "ab9bf10abba79478c607a6bf13d95938"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.454838 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bdafe788482b4fd8a04f287f3d289883"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "944ddaac54499732be9580bd3915790e"
server_key_iv: "ab9bf10abba79478c607a6bf13d95938"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.458452 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:19.460918 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.462244 3640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:19.462373 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "bdafe788482b4fd8a04f287f3d289883"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "944ddaac54499732be9580bd3915790e"
server_key_iv: "ab9bf10abba79478c607a6bf13d95938"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.462481 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:19.477156 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:19.480201 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:19.480410 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:19.480979 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:19.481758 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:19.481825 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.481890 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:19.481940 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.490747 3640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37545
I20260929 18:31:19.490789 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37545 every 8 connection(s)
I20260929 18:31:19.491616 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:19.500720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3640
I20260929 18:31:19.500830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:19.501114 392 external_mini_cluster.cc:1428] Setting key be67f0867e63bd1894bfaa97133f5324
I20260929 18:31:19.505681 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.493443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41813 (local address 127.0.98.62:33197)
0929 18:31:19.493591 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:19.493595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:19.494473 (+ 878us) server_negotiation.cc:407] Connection header received
0929 18:31:19.495784 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:19.495787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:19.495831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:19.495912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:19.497080 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.497616 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.498797 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.499018 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.499101 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.501654 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:19.501660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:19.501667 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:19.501696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:19.503549 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.504087 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.504093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.504098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.504222 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.504604 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.504609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.504611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.504803 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:19.504909 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:19.505399 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 18:31:19.505517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:19.506460 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.493876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33197 (local address 127.0.98.2:41813)
0929 18:31:19.494318 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:19.494359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:19.495565 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:19.496068 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:19.496075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:19.496479 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:19.496927 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.496939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.497777 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.497781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.498653 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.498662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.499159 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.499162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.499319 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.499849 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:19.499865 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:19.501491 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:19.503673 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.503678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.503689 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.503985 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.504334 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.504337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.504338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.504496 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.504918 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:19.504928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:19.505158 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:31:19.505346 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 18:31:19.505665 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260929 18:31:19.507581 3769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33197
I20260929 18:31:19.507843 3769 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:19.508481 3769 heartbeater.cc:511] Master 127.0.98.62:33197 requested a full tablet report, sending...
I20260929 18:31:19.509603 3444 ts_manager.cc:194] Registered new tserver with Master: bdafe788482b4fd8a04f287f3d289883 (127.0.98.2:37545)
I20260929 18:31:19.510164 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41813
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:19.528512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:19.634272 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:19.634563 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:19.634671 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:19.638752 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:19.638839 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:19.638947 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:19.644241 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:19.645449 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:19.646533 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:19.656517 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:19.656665 3774 server_base.cc:1034] running on GCE node
W20260929 18:31:19.656522 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:19.656522 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:19.657222 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:19.657912 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:19.659071 3774 hybrid_clock.cc:648] HybridClock initialized: now 1790706679659037 us; error 48 us; skew 500 ppm
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:19.661963 3774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:19.662810 3774 webserver.cc:460] Webserver started at http://127.0.98.3:33235/ using document root <none> and password file <none>
I20260929 18:31:19.663391 3774 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:19.663479 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:19.663700 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:19.665663 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "f887dcbc5f8c4c1f8e9e22b09461d59d"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "bab35f6e5b8c6b1a087ecf3ba1885edf"
server_key_iv: "5dd52bf456dc0430d438521ff1ab4594"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.666278 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f887dcbc5f8c4c1f8e9e22b09461d59d"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "bab35f6e5b8c6b1a087ecf3ba1885edf"
server_key_iv: "5dd52bf456dc0430d438521ff1ab4594"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.669809 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:19.672071 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.673269 3774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:19.673408 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "f887dcbc5f8c4c1f8e9e22b09461d59d"
format_stamp: "Formatted at 2026-09-29 18:31:19 on dist-test-slave-fmv8"
server_key: "bab35f6e5b8c6b1a087ecf3ba1885edf"
server_key_iv: "5dd52bf456dc0430d438521ff1ab4594"
server_key_version: "encryptionkey@0"
I20260929 18:31:19.673537 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:19.685899 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:19.689013 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:19.689205 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:19.689749 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:19.690515 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:19.690585 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.690655 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:19.690696 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:19.699213 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43489
I20260929 18:31:19.699254 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43489 every 8 connection(s)
I20260929 18:31:19.700038 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:19.704046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3774
I20260929 18:31:19.704195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:19.704499 392 external_mini_cluster.cc:1428] Setting key 9099754471a641302254e5118ba274f5
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:19.713656 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.701741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56751 (local address 127.0.98.62:33197)
0929 18:31:19.701900 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:19.701904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:19.702630 (+ 726us) server_negotiation.cc:407] Connection header received
0929 18:31:19.704000 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:19.704004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:19.704056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:19.704163 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:19.705711 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.706473 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.707588 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.707824 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.707938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.710418 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:19.710421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:19.710423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:19.710446 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:19.711710 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.712311 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.712314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.712316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.712396 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.712729 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.712731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.712733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.712919 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:19.712993 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:19.713446 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 18:31:19.713541 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:31:19.714530 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.702086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33197 (local address 127.0.98.3:56751)
0929 18:31:19.702495 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:19.702533 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:19.703804 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:19.704397 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:19.704407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:19.704872 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:19.705548 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.705570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.706601 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.706604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.707466 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:19.707474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.707971 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.707975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:19.708198 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.708814 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:19.708830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:19.710234 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:19.711835 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.711846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.711857 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.712104 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.712507 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:19.712509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:19.712510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:19.712639 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:19.713010 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:19.713014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:19.713233 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 18:31:19.713385 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:31:19.713623 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260929 18:31:19.715700 3903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33197
I20260929 18:31:19.715936 3903 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:19.716468 3903 heartbeater.cc:511] Master 127.0.98.62:33197 requested a full tablet report, sending...
I20260929 18:31:19.717495 3440 ts_manager.cc:194] Registered new tserver with Master: f887dcbc5f8c4c1f8e9e22b09461d59d (127.0.98.3:43489)
I20260929 18:31:19.718084 3440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56751
I20260929 18:31:19.718211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706679, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:19.742789 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:19.734300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.98.62:33197)
0929 18:31:19.734421 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:19.734425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:19.734616 (+ 191us) server_negotiation.cc:407] Connection header received
0929 18:31:19.734707 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:19.734709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:19.734745 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:19.734817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:19.735496 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.736015 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.736952 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:19.737160 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:19.737257 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:19.739745 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:19.739749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:19.739751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:19.739782 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:19.741264 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.741652 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.741655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.741656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.741781 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:19.742017 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:19.742019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:19.742020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:19.742226 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:19.742330 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:19.742546 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:31:19.742660 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:31:19.743170 3440 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60324
I20260929 18:31:19.744202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3506
I20260929 18:31:19.749440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3640
I20260929 18:31:19.754642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3774
I20260929 18:31:19.759509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3412
2026-09-29T18:31:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2101 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3915](info): set up 2 sockets
Sep 29 18:31:19 dist-test-slave-fmv8 krb5kdc[3915](info): commencing operation
krb5kdc: starting...
W20260929 18:31:20.796070 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:20 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:20.913445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:21.020974 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:21.021239 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:21.021279 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:21.025169 3928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:21.025216 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:21.025235 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:21.025251 3928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:21.025266 3928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:21.030376 3928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:21.031312 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:21.032294 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:21.040434 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.040484 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.040483 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:21.040643 3928 server_base.cc:1034] running on GCE node
I20260929 18:31:21.041262 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:21.042330 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:21.043517 3928 hybrid_clock.cc:648] HybridClock initialized: now 1790706681043487 us; error 42 us; skew 500 ppm
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:21.046430 3928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:21.047242 3928 webserver.cc:460] Webserver started at http://127.0.98.62:40275/ using document root <none> and password file <none>
I20260929 18:31:21.047739 3928 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:21.047801 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:21.047960 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:21.049926 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "a19227de098045709d1d019fbd0d5f8d"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "e7a1ba1524dc42bc3d0a8f90ea772f96"
server_key_iv: "0f26675aeefe8da355f8f524709c5db1"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.050608 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "a19227de098045709d1d019fbd0d5f8d"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "e7a1ba1524dc42bc3d0a8f90ea772f96"
server_key_iv: "0f26675aeefe8da355f8f524709c5db1"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.054360 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:21.056763 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.057919 3928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:21.058054 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "a19227de098045709d1d019fbd0d5f8d"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "e7a1ba1524dc42bc3d0a8f90ea772f96"
server_key_iv: "0f26675aeefe8da355f8f524709c5db1"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.058185 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:21.080027 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:21.083410 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:21.083611 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:21.090415 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39223
I20260929 18:31:21.090479 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39223 every 8 connection(s)
I20260929 18:31:21.091390 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:21.094288 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:21.099217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 3928
I20260929 18:31:21.099356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:21.099669 392 external_mini_cluster.cc:1428] Setting key cd8b903f0ef668961720a5bac05d05bc
I20260929 18:31:21.100082 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: Bootstrap starting.
I20260929 18:31:21.102564 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:21.103523 3996 log.cc:824] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:21.105780 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: No bootstrap required, opened a new log
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:21.110209 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } }
I20260929 18:31:21.110472 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:21.110553 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19227de098045709d1d019fbd0d5f8d, State: Initialized, Role: FOLLOWER
I20260929 18:31:21.111061 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } }
I20260929 18:31:21.111228 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:21.111325 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:21.111442 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:21.112428 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } }
I20260929 18:31:21.112820 3996 leader_election.cc:302] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19227de098045709d1d019fbd0d5f8d; no voters:
I20260929 18:31:21.113214 3996 leader_election.cc:288] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:21.113343 4001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:21.113588 4001 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [term 1 LEADER]: Becoming Leader. State: Replica: a19227de098045709d1d019fbd0d5f8d, State: Running, Role: LEADER
I20260929 18:31:21.113971 4001 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } }
I20260929 18:31:21.114688 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:21.115855 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19227de098045709d1d019fbd0d5f8d. Latest consensus state: current_term: 1 leader_uuid: "a19227de098045709d1d019fbd0d5f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } } }
I20260929 18:31:21.115974 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:21.116111 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.101156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.0.98.62:39223)
0929 18:31:21.101577 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.101599 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.101651 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:31:21.102296 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.102313 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.102699 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.103004 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.103946 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.105087 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.106283 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.106756 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.106875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.109918 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.109923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.109937 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.109964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.112118 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.112707 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.112714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.112719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.112864 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.113197 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.113201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.113203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.113759 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.113983 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.114638 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.114962 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":145,"threads_started":1}
I20260929 18:31:21.116601 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19227de098045709d1d019fbd0d5f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19227de098045709d1d019fbd0d5f8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39223 } } }
I20260929 18:31:21.117650 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d [sys.catalog]: This master's current role is: LEADER
W20260929 18:31:21.119690 4016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:21.119755 4016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:21.119854 4010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:21.120785 4010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:21.126226 4010 catalog_manager.cc:1347] Generated new cluster ID: 7d2e83cd05b64252a69409c175e1729c
I20260929 18:31:21.126286 4010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:21.143738 4010 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:21.144814 4010 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:21.149919 4010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19227de098045709d1d019fbd0d5f8d: Generated new TSK 0
I20260929 18:31:21.150751 4010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:21.186841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:21.297109 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:21.297355 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:21.297397 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:21.301334 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:21.301388 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:21.301442 4022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:21.307184 4022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:21.308112 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:21.309160 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:21.318475 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.318464 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.318487 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:21.318835 4022 server_base.cc:1034] running on GCE node
I20260929 18:31:21.319205 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:21.319845 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:21.321041 4022 hybrid_clock.cc:648] HybridClock initialized: now 1790706681321002 us; error 58 us; skew 500 ppm
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:21.323879 4022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:21.324795 4022 webserver.cc:460] Webserver started at http://127.0.98.1:43477/ using document root <none> and password file <none>
I20260929 18:31:21.325356 4022 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:21.325448 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:21.325675 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:21.327658 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "f27fc46a14dd4d0bb41b92d4f7dd9c7e"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "f36f08bf99a5dc0c977c3235ac7bf08d"
server_key_iv: "6a3b2942e354072da5f0254af1952b19"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.328282 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f27fc46a14dd4d0bb41b92d4f7dd9c7e"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "f36f08bf99a5dc0c977c3235ac7bf08d"
server_key_iv: "6a3b2942e354072da5f0254af1952b19"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.331844 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:21.334295 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.335726 4022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:21.335860 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "f27fc46a14dd4d0bb41b92d4f7dd9c7e"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "f36f08bf99a5dc0c977c3235ac7bf08d"
server_key_iv: "6a3b2942e354072da5f0254af1952b19"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.335979 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:21.358166 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:21.361325 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:21.361572 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:21.362152 4022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:21.363057 4022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:21.363103 4022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.363142 4022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:21.363157 4022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.371591 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35011
I20260929 18:31:21.371613 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35011 every 8 connection(s)
I20260929 18:31:21.372679 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:21.373245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4022
I20260929 18:31:21.373368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:21.373662 392 external_mini_cluster.cc:1428] Setting key d9452295b38ff626bd56181f8651daa7
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:21.392138 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.374437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42999 (local address 127.0.98.62:39223)
0929 18:31:21.374567 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.374571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.375410 (+ 839us) server_negotiation.cc:407] Connection header received
0929 18:31:21.377165 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.377168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.377210 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.377282 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.378896 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.379452 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.381358 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.381764 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.381858 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.387770 (+ 5912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.387774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.387776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.387802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.389685 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.390257 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.390264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.390269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.390371 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.390769 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.390775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.390779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.391043 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.391154 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.391726 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.391898 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:31:21.392925 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.374827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39223 (local address 127.0.98.1:42999)
0929 18:31:21.375253 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:21.375304 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:21.376926 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:21.377495 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:21.377507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:21.378028 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:21.378709 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.378731 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.379584 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.379588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.381208 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.381222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.382074 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.382081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.382282 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.382966 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:21.382994 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:21.387543 (+ 4549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:21.389821 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.389826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.389839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.390128 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.390491 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.390496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.390498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.390655 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.391178 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:21.391184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:21.391435 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 18:31:21.391648 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 18:31:21.391936 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260929 18:31:21.394223 4151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39223
I20260929 18:31:21.394512 4151 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:21.395118 4151 heartbeater.cc:511] Master 127.0.98.62:39223 requested a full tablet report, sending...
I20260929 18:31:21.396827 3960 ts_manager.cc:194] Registered new tserver with Master: f27fc46a14dd4d0bb41b92d4f7dd9c7e (127.0.98.1:35011)
I20260929 18:31:21.397970 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:21.403687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:21.512948 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:21.513199 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:21.513240 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:21.517493 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:21.517549 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:21.517606 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:21.522673 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:21.523582 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:21.524549 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:21.533808 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:21.533993 4156 server_base.cc:1034] running on GCE node
W20260929 18:31:21.533807 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.533825 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:21.534497 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:21.535157 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:21.536342 4156 hybrid_clock.cc:648] HybridClock initialized: now 1790706681536328 us; error 26 us; skew 500 ppm
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:21.539001 4156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:21.539887 4156 webserver.cc:460] Webserver started at http://127.0.98.2:41323/ using document root <none> and password file <none>
I20260929 18:31:21.540485 4156 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:21.540575 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:21.540807 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:21.542754 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "0df6db68eb0d421ca1d40e4a8043afae"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1dd1cd76d86355e08a92d790ddad6919"
server_key_iv: "1325fa6c1aa25b5ce1098935b8ec234e"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.543299 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0df6db68eb0d421ca1d40e4a8043afae"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1dd1cd76d86355e08a92d790ddad6919"
server_key_iv: "1325fa6c1aa25b5ce1098935b8ec234e"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.546931 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 18:31:21.549299 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.550426 4156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:21.550560 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "0df6db68eb0d421ca1d40e4a8043afae"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1dd1cd76d86355e08a92d790ddad6919"
server_key_iv: "1325fa6c1aa25b5ce1098935b8ec234e"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.550669 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:21.560371 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:21.563640 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:21.563834 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:21.564463 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:21.565274 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:21.565325 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.565392 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:21.565435 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.574054 4156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36103
I20260929 18:31:21.574110 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36103 every 8 connection(s)
I20260929 18:31:21.574918 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:21.579550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4156
I20260929 18:31:21.579674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:21.579982 392 external_mini_cluster.cc:1428] Setting key 37fbe75cf2497fcaa0b8fdbaf7874333
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:21.589138 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.576652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45157 (local address 127.0.98.62:39223)
0929 18:31:21.576784 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.576788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.577608 (+ 820us) server_negotiation.cc:407] Connection header received
0929 18:31:21.579000 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.579004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.579069 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.579160 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.580492 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.581129 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.582327 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.582725 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.582840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.585774 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.585777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.585779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.585802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.587178 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.587666 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.587672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.587673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.587748 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.588067 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.588072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.588074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.588304 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.588407 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.588862 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.588973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:21.589882 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.576987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39223 (local address 127.0.98.2:45157)
0929 18:31:21.577426 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:21.577466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:21.578775 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:21.579338 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:21.579345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:21.579767 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:21.580353 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.580369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.581269 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.581273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.582202 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.582211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.582918 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.582923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.583161 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.583870 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:21.583895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:21.585610 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:21.587301 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.587306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.587320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.587561 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.587844 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.587847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.587849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.587973 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.588404 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:21.588410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:21.588640 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:31:21.588821 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:31:21.589127 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260929 18:31:21.591262 4285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39223
I20260929 18:31:21.591539 4285 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:21.592130 4285 heartbeater.cc:511] Master 127.0.98.62:39223 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:31:21.593242 3960 ts_manager.cc:194] Registered new tserver with Master: 0df6db68eb0d421ca1d40e4a8043afae (127.0.98.2:36103)
I20260929 18:31:21.594084 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:21.608912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:21.718250 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:21.718490 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:21.718576 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:21.722685 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:21.722743 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:21.722827 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:21.727950 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:21.728966 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:21.730070 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:21.739650 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.739578 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:21.739669 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:21.739871 4290 server_base.cc:1034] running on GCE node
I20260929 18:31:21.740433 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:21.741140 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:21.742311 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790706681742290 us; error 32 us; skew 500 ppm
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:21.745324 4290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:21.746177 4290 webserver.cc:460] Webserver started at http://127.0.98.3:35315/ using document root <none> and password file <none>
I20260929 18:31:21.746726 4290 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:21.746815 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:21.747041 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:21.749106 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "149235299c6c416fb09758c2c5a0282c"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1a40fa4eeaab8fbfcedda8e2c228803e"
server_key_iv: "4b3e393a8108d83d6e44ed2045748266"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.749652 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "149235299c6c416fb09758c2c5a0282c"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1a40fa4eeaab8fbfcedda8e2c228803e"
server_key_iv: "4b3e393a8108d83d6e44ed2045748266"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.753214 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:21.755710 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.757091 4290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:21.757247 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "149235299c6c416fb09758c2c5a0282c"
format_stamp: "Formatted at 2026-09-29 18:31:21 on dist-test-slave-fmv8"
server_key: "1a40fa4eeaab8fbfcedda8e2c228803e"
server_key_iv: "4b3e393a8108d83d6e44ed2045748266"
server_key_version: "encryptionkey@0"
I20260929 18:31:21.757361 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:21.767732 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:21.770750 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:21.770943 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:21.771497 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:21.772352 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:21.772419 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.772482 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:21.772531 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:21.781468 4290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40857
I20260929 18:31:21.781493 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40857 every 8 connection(s)
I20260929 18:31:21.782333 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:21.784523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4290
I20260929 18:31:21.784637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:21.784934 392 external_mini_cluster.cc:1428] Setting key 306ad064c081a595e4f782c8e802aa14
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:21.796702 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.784292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52859 (local address 127.0.98.62:39223)
0929 18:31:21.784450 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.784453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.785481 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 18:31:21.787367 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.787369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.787412 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.787475 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.788700 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.789235 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.790417 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.790643 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.790725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.793284 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.793286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.793288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.793313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.794760 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.795237 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.795239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.795240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.795311 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.795630 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.795632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.795633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.795810 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.795884 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.796470 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.796569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:31:21.797513 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.784792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39223 (local address 127.0.98.3:52859)
0929 18:31:21.785331 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:21.785388 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:21.787167 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:21.787642 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:21.787648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:21.788025 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:21.788545 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.788559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.789381 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.789384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.790296 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:21.790306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.790781 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.790783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:21.790924 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.791597 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:21.791614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:21.793124 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:21.794869 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.794874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.794884 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.795141 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.795390 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:21.795392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:21.795393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:21.795523 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:21.795923 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:21.795928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:21.796248 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 18:31:21.796411 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:31:21.796686 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260929 18:31:21.798710 4419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39223
I20260929 18:31:21.798952 4419 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:21.799436 4419 heartbeater.cc:511] Master 127.0.98.62:39223 requested a full tablet report, sending...
I20260929 18:31:21.800443 3960 ts_manager.cc:194] Registered new tserver with Master: 149235299c6c416fb09758c2c5a0282c (127.0.98.3:40857)
I20260929 18:31:21.800994 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52859
I20260929 18:31:21.810770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:21.836736 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.827101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.0.98.1:35011)
0929 18:31:21.827383 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.827396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.827428 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:31:21.827480 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.827485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.827673 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.827796 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.828538 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.829327 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.830199 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.830611 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.830699 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.833188 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.833197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.833209 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.833233 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.834797 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.835257 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.835264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.835265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.835343 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.835634 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.835637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.835639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.835877 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.836128 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.836404 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.836559 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:21.847719 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:21.839465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.62:39223)
0929 18:31:21.839590 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:21.839593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:21.839743 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:31:21.839819 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:21.839821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:21.839866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:21.840013 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:21.840731 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.841251 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.842064 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:21.842277 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:21.842387 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:21.844732 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:21.844736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:21.844739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:21.844770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:21.846035 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.846450 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.846454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.846456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.846547 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:21.846850 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:21.846855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:21.846857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:21.847071 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:21.847208 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:21.847425 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:31:21.847542 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:31:21.848097 3960 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45474
I20260929 18:31:21.848743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4022
I20260929 18:31:21.854478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4156
I20260929 18:31:21.859696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4290
I20260929 18:31:21.865181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 3928
2026-09-29T18:31:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2105 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[4431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[4431](info): set up 2 sockets
Sep 29 18:31:21 dist-test-slave-fmv8 krb5kdc[4431](info): commencing operation
krb5kdc: starting...
W20260929 18:31:22.908353 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:22 dist-test-slave-fmv8 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:23.029047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:23.135149 4444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:23.135399 4444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:23.135442 4444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:23.139472 4444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:23.139528 4444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:23.139551 4444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:23.139568 4444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:23.139622 4444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:23.144780 4444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:23.145812 4444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:23.146799 4444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:23.154955 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:23.154996 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.155062 4444 server_base.cc:1034] running on GCE node
W20260929 18:31:23.155162 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.155823 4444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:23.157011 4444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:23.158203 4444 hybrid_clock.cc:648] HybridClock initialized: now 1790706683158191 us; error 27 us; skew 500 ppm
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:23.168900 4444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:23.169605 4444 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:23.169669 4444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:23.169829 4444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:23.171808 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "29d7f71ab88a4449ab9f572c1a7da841"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "bd6801943f7963d5388fb0988ba298af"
server_key_iv: "f3ff58e9f1a395a1b0dcf39566b33028"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.172392 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "29d7f71ab88a4449ab9f572c1a7da841"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "bd6801943f7963d5388fb0988ba298af"
server_key_iv: "f3ff58e9f1a395a1b0dcf39566b33028"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.175961 4444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:31:23.178506 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.179750 4444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:23.179849 4444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "29d7f71ab88a4449ab9f572c1a7da841"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "bd6801943f7963d5388fb0988ba298af"
server_key_iv: "f3ff58e9f1a395a1b0dcf39566b33028"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.179931 4444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:23.194271 4444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:23.197543 4444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:23.197688 4444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:23.204651 4444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33889
I20260929 18:31:23.204933 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33889 every 8 connection(s)
I20260929 18:31:23.205344 4444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:23.207862 4511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:23.213044 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: Bootstrap starting.
I20260929 18:31:23.214933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4444
I20260929 18:31:23.215032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:23.215328 392 external_mini_cluster.cc:1428] Setting key 97422bbe155349ff12a59ab2a188b285
I20260929 18:31:23.215461 4511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:23.216464 4511 log.cc:824] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:23.218420 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: No bootstrap required, opened a new log
I20260929 18:31:23.221876 4511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260929 18:31:23.222133 4511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:23.222203 4511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d7f71ab88a4449ab9f572c1a7da841, State: Initialized, Role: FOLLOWER
I20260929 18:31:23.222700 4511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260929 18:31:23.222868 4511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:23.222947 4511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:23.223066 4511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:23.224211 4511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260929 18:31:23.224591 4511 leader_election.cc:302] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d7f71ab88a4449ab9f572c1a7da841; no voters:
I20260929 18:31:23.224948 4511 leader_election.cc:288] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:23.225054 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:23.225282 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [term 1 LEADER]: Becoming Leader. State: Replica: 29d7f71ab88a4449ab9f572c1a7da841, State: Running, Role: LEADER
I20260929 18:31:23.225651 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260929 18:31:23.226325 4511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:23.227320 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d7f71ab88a4449ab9f572c1a7da841. Latest consensus state: current_term: 1 leader_uuid: "29d7f71ab88a4449ab9f572c1a7da841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } } }
I20260929 18:31:23.227444 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:23.227763 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d7f71ab88a4449ab9f572c1a7da841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d7f71ab88a4449ab9f572c1a7da841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } } }
I20260929 18:31:23.227849 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:23.228262 4525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:23.230691 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.216759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.98.62:33889)
0929 18:31:23.217201 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.217216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.217250 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:31:23.217837 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.217852 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.218180 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.218502 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:23.219279 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.220296 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.221150 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.221608 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.221783 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.224681 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:23.224686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:23.224720 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:23.224756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:23.226905 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.227473 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.227479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.227484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.227626 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.227957 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.227961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.227962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.228489 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:23.228671 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.229512 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.229794 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260929 18:31:23.230924 4525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:23.236709 4525 catalog_manager.cc:1347] Generated new cluster ID: 2d2ba6dde63e4205997cd45c3b08146e
I20260929 18:31:23.236786 4525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:23.258104 4525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:23.259371 4525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:23.268327 4525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: Generated new TSK 0
I20260929 18:31:23.269168 4525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:23.303107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:23.410243 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:23.410527 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:23.410604 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:23.414542 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:23.414633 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:23.414724 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:23.419740 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:23.420779 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:23.421777 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:23.431012 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.430992 4537 server_base.cc:1034] running on GCE node
W20260929 18:31:23.430986 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:23.431039 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.431725 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:23.432443 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:23.433660 4537 hybrid_clock.cc:648] HybridClock initialized: now 1790706683433650 us; error 31 us; skew 500 ppm
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:23.436610 4537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:23.437372 4537 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:23.437465 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:23.437690 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:23.439651 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "0804c8b116344914b650c9d43610b705"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "a782cb5cbfcf78c5bd47198304631c1c"
server_key_iv: "9612c04ce50429bebcb1b8f03c447181"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.440316 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0804c8b116344914b650c9d43610b705"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "a782cb5cbfcf78c5bd47198304631c1c"
server_key_iv: "9612c04ce50429bebcb1b8f03c447181"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.443904 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:23.446480 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.447719 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:23.447866 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "0804c8b116344914b650c9d43610b705"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "a782cb5cbfcf78c5bd47198304631c1c"
server_key_iv: "9612c04ce50429bebcb1b8f03c447181"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.447976 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:23.458969 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:23.462085 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:23.462307 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:23.462894 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:23.463753 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:23.463830 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.463896 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:23.463953 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.473798 4537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32899
I20260929 18:31:23.473824 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32899 every 8 connection(s)
I20260929 18:31:23.474395 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:23.479234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4537
I20260929 18:31:23.479342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:23.479713 392 external_mini_cluster.cc:1428] Setting key 8da8e17695e552ef976d33a92e493636
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:23.489389 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.476085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53333 (local address 127.0.98.62:33889)
0929 18:31:23.476288 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.476294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.477193 (+ 899us) server_negotiation.cc:407] Connection header received
0929 18:31:23.478517 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.478520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.478570 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.478649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:23.479983 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.480677 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.482333 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.482560 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.482725 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.485370 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:23.485375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:23.485378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:23.485408 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:23.487172 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.487717 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.487722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.487724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.487829 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.488192 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.488195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.488197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.488442 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:23.488551 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.489095 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.489225 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:23.490167 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.476537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33889 (local address 127.0.98.1:53333)
0929 18:31:23.477040 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.477080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.478301 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.478809 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.478816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.479196 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:23.479835 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.479857 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.480798 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.480802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.482145 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.482159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.482765 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.482772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.482938 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.483548 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:23.483564 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:23.485184 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:23.487323 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.487331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.487347 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.487609 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.487926 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.487928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.487929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.488073 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.488574 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:23.488579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:23.488831 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.489004 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.489263 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
I20260929 18:31:23.491590 4665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:31:23.491861 4665 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:23.492422 4665 heartbeater.cc:511] Master 127.0.98.62:33889 requested a full tablet report, sending...
I20260929 18:31:23.494092 4475 ts_manager.cc:194] Registered new tserver with Master: 0804c8b116344914b650c9d43610b705 (127.0.98.1:32899)
I20260929 18:31:23.495186 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:23.506727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:23.613893 4670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:23.614173 4670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:23.614250 4670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:23.618319 4670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:23.618403 4670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:23.618494 4670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:23.623584 4670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:23.624641 4670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:23.625617 4670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:23.635016 4675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.635134 4670 server_base.cc:1034] running on GCE node
W20260929 18:31:23.635027 4676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:23.635325 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.635757 4670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:23.636392 4670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:23.637596 4670 hybrid_clock.cc:648] HybridClock initialized: now 1790706683637561 us; error 47 us; skew 500 ppm
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:23.640323 4670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:23.640975 4670 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:23.641036 4670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:23.641191 4670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:23.643144 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a401541cb0c45dab919bc3ced47f6df"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "ea23c84f7b10293100cdfd2fd203f77d"
server_key_iv: "4d984acd11bb443670263ee57f619f0a"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.643631 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a401541cb0c45dab919bc3ced47f6df"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "ea23c84f7b10293100cdfd2fd203f77d"
server_key_iv: "4d984acd11bb443670263ee57f619f0a"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.647128 4670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:23.649689 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.651042 4670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:23.651136 4670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "2a401541cb0c45dab919bc3ced47f6df"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "ea23c84f7b10293100cdfd2fd203f77d"
server_key_iv: "4d984acd11bb443670263ee57f619f0a"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.651235 4670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:23.663546 4670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:23.666599 4670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:23.666745 4670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:23.667327 4670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:23.668126 4670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:23.668242 4670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.668316 4670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:23.668350 4670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.676994 4670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42465
I20260929 18:31:23.677063 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42465 every 8 connection(s)
I20260929 18:31:23.677600 4670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:23.682565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4670
I20260929 18:31:23.682700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:23.682963 392 external_mini_cluster.cc:1428] Setting key c009e265513a031b2ae7d705f829dd57
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:23.692286 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.679364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38709 (local address 127.0.98.62:33889)
0929 18:31:23.679525 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.679528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.680368 (+ 840us) server_negotiation.cc:407] Connection header received
0929 18:31:23.681748 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.681752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.681805 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.681914 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:23.683343 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.683973 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.685298 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.685522 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.685631 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.688496 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:23.688499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:23.688501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:23.688546 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:23.690045 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.690657 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.690659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.690660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.690730 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.691120 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.691122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.691123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.691299 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:23.691420 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.692002 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.692097 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:23.693367 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.679752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33889 (local address 127.0.98.2:38709)
0929 18:31:23.680232 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.680273 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.681545 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.682103 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.682109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.682606 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:23.683198 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.683214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.684097 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.684101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.685165 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.685174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.685667 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.685671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.685848 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.686477 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:23.686499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:23.688315 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:23.690168 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.690174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.690190 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.690550 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.690820 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.690824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.690827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.691022 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.691401 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:23.691412 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:23.691728 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.691966 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.692359 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260929 18:31:23.694478 4798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33889
I20260929 18:31:23.694736 4798 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:31:23.695279 4798 heartbeater.cc:511] Master 127.0.98.62:33889 requested a full tablet report, sending...
I20260929 18:31:23.696516 4475 ts_manager.cc:194] Registered new tserver with Master: 2a401541cb0c45dab919bc3ced47f6df (127.0.98.2:42465)
I20260929 18:31:23.697084 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:23.709403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:23.819833 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:23.820133 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:23.820252 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:23.824221 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:23.824309 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:23.824399 4803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:23.829705 4803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:23.830674 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:23.831657 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:23.840515 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:23.840510 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:23.840510 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:23.840555 4803 server_base.cc:1034] running on GCE node
I20260929 18:31:23.841096 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:23.841744 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:23.842913 4803 hybrid_clock.cc:648] HybridClock initialized: now 1790706683842902 us; error 26 us; skew 500 ppm
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:23.845698 4803 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:23.846702 4803 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:23.846812 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:23.847046 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:23.849103 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "afa050f254f44c099a8170f3cfc4145d"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "4cce63183d4b800afe9374f3e28b923f"
server_key_iv: "76b82389c05252410209a9625973e2c4"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.849668 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "afa050f254f44c099a8170f3cfc4145d"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "4cce63183d4b800afe9374f3e28b923f"
server_key_iv: "76b82389c05252410209a9625973e2c4"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.853294 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:23.855873 4817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.857147 4803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:23.857285 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "afa050f254f44c099a8170f3cfc4145d"
format_stamp: "Formatted at 2026-09-29 18:31:23 on dist-test-slave-fmv8"
server_key: "4cce63183d4b800afe9374f3e28b923f"
server_key_iv: "76b82389c05252410209a9625973e2c4"
server_key_version: "encryptionkey@0"
I20260929 18:31:23.857393 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:23.867334 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:23.870119 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:23.870307 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:23.870833 4803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:23.871609 4803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:23.871680 4803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.871745 4803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:23.871794 4803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:23.880770 4803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44149
I20260929 18:31:23.880805 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44149 every 8 connection(s)
I20260929 18:31:23.881333 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:23.885578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4803
I20260929 18:31:23.885704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:23.885975 392 external_mini_cluster.cc:1428] Setting key 66e449321761aa20d4b95ed9c8a1b815
Sep 29 18:31:23 dist-test-slave-fmv8 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:23.894976 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.883132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43625 (local address 127.0.98.62:33889)
0929 18:31:23.883267 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.883271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.884183 (+ 912us) server_negotiation.cc:407] Connection header received
0929 18:31:23.885563 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.885566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.885620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.885706 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:23.887127 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.887653 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.888810 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.889071 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.889183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.891562 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:23.891565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:23.891567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:23.891590 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:23.893141 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.893633 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.893636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.893637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.893713 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.894035 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.894038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.894039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.894212 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:23.894290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.894748 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.894860 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:31:23.895740 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.883538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33889 (local address 127.0.98.3:43625)
0929 18:31:23.884011 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.884063 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.885355 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.885883 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.885890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.886403 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:23.886985 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.887001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.887776 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.887780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.888681 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.888690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.889235 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.889239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.889380 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.890010 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:23.890026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:23.891400 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:23.893251 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.893255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.893266 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.893532 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.893799 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:23.893801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:23.893803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:23.893940 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:23.894298 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:23.894303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:23.894532 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.894683 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.894943 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260929 18:31:23.896898 4931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33889
I20260929 18:31:23.897133 4931 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:23.897563 4931 heartbeater.cc:511] Master 127.0.98.62:33889 requested a full tablet report, sending...
I20260929 18:31:23.898617 4475 ts_manager.cc:194] Registered new tserver with Master: afa050f254f44c099a8170f3cfc4145d (127.0.98.3:44149)
I20260929 18:31:23.899300 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43625
I20260929 18:31:23.899880 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:31:23.909268 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.902015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.0.98.62:33889)
0929 18:31:23.902123 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.902126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.902289 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:31:23.902445 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.902447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.902479 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.902538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:23.903240 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.903719 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.904664 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.904880 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.904989 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.906274 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:23.906277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:23.906279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:23.906298 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:23.907748 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.908217 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.908221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.908222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.908298 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:23.908556 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:23.908559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:23.908560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:23.908767 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:23.908872 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.909068 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.909159 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:23.912391 4474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:23.915330 4474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:23.930480 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.924613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44149 (local address 127.0.0.1:33422)
0929 18:31:23.925481 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.925495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.925572 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.926096 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.926101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.926121 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:23.926214 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.926233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.927480 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.927486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.928893 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.928902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.929930 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.929934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.930042 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930082 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.930196 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.930324 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":90,"thread_start_us":80,"threads_started":1}
I20260929 18:31:23.930632 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.924410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.0.98.1:32899)
0929 18:31:23.925329 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.925344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.925367 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:31:23.925489 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.925496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.925710 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.925856 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:23.926181 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.927360 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.929232 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.930023 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.930128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930250 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.930384 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.930474 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":105,"threads_started":1}
I20260929 18:31:23.931104 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.925016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.98.3:44149)
0929 18:31:23.925633 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.925644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.925664 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:31:23.925731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.925736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.925958 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.926105 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:23.926344 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.927349 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.929012 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.929802 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.929929 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930075 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.930874 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.930961 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":160,"threads_started":1}
I20260929 18:31:23.931104 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.925056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.0.98.2:42465)
0929 18:31:23.926168 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.926179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.926198 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:31:23.926296 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.926303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.926487 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.926633 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:23.926873 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.928173 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.929504 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.930140 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.930232 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930354 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.930878 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.930961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":127,"threads_started":1}
I20260929 18:31:23.930548 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.924224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32899 (local address 127.0.0.1:44534)
0929 18:31:23.924878 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.924927 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.925053 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.925886 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.925893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.925916 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:23.926029 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.926033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.927503 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.927506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.929105 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.929115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.930150 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.930154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.930240 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930266 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.930338 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.930408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":79,"thread_start_us":89,"threads_started":1}
I20260929 18:31:23.930799 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.924390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42465 (local address 127.0.0.1:50948)
0929 18:31:23.925096 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.925110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.925199 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.926628 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.926631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.926648 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:23.926723 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.926728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.928327 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.928337 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.929389 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.929396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.930410 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.930414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.930518 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.930550 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.930626 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.930686 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":99,"threads_started":1}
I20260929 18:31:23.934223 4865 tablet_service.cc:1467] Processing CreateTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 (DEFAULT_TABLE table=test-table [id=cdbf00920c914d55975ce03247c9e1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:23.934257 4599 tablet_service.cc:1467] Processing CreateTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 (DEFAULT_TABLE table=test-table [id=cdbf00920c914d55975ce03247c9e1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:23.934257 4732 tablet_service.cc:1467] Processing CreateTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 (DEFAULT_TABLE table=test-table [id=cdbf00920c914d55975ce03247c9e1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:23.935209 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301c60efd9674678bd68c2ec19f24cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:23.935220 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301c60efd9674678bd68c2ec19f24cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:23.935220 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301c60efd9674678bd68c2ec19f24cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:23.942863 4948 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Bootstrap starting.
I20260929 18:31:23.942942 4949 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Bootstrap starting.
I20260929 18:31:23.944473 4950 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Bootstrap starting.
I20260929 18:31:23.944707 4948 tablet_bootstrap.cc:654] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:23.944739 4949 tablet_bootstrap.cc:654] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:23.945740 4948 log.cc:824] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:23.946066 4949 log.cc:824] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:23.946624 4950 tablet_bootstrap.cc:654] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:23.947701 4948 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: No bootstrap required, opened a new log
I20260929 18:31:23.947718 4950 log.cc:824] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:23.947896 4948 ts_tablet_manager.cc:1397] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:31:23.949330 4950 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: No bootstrap required, opened a new log
I20260929 18:31:23.949534 4950 ts_tablet_manager.cc:1397] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:31:23.950910 4948 raft_consensus.cc:348] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.951164 4948 raft_consensus.cc:374] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:23.951244 4948 raft_consensus.cc:729] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa050f254f44c099a8170f3cfc4145d, State: Initialized, Role: FOLLOWER
I20260929 18:31:23.951767 4948 consensus_queue.cc:260] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.952636 4931 heartbeater.cc:503] Master 127.0.98.62:33889 was elected leader, sending a full tablet report...
I20260929 18:31:23.952783 4948 ts_tablet_manager.cc:1428] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:31:23.952976 4949 tablet_bootstrap.cc:492] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: No bootstrap required, opened a new log
I20260929 18:31:23.953148 4949 ts_tablet_manager.cc:1397] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 18:31:23.953048 4950 raft_consensus.cc:348] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.953282 4950 raft_consensus.cc:374] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:23.953336 4950 raft_consensus.cc:729] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0804c8b116344914b650c9d43610b705, State: Initialized, Role: FOLLOWER
I20260929 18:31:23.953714 4950 consensus_queue.cc:260] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.954353 4665 heartbeater.cc:503] Master 127.0.98.62:33889 was elected leader, sending a full tablet report...
I20260929 18:31:23.954622 4950 ts_tablet_manager.cc:1428] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:31:23.956915 4949 raft_consensus.cc:348] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.957193 4949 raft_consensus.cc:374] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:23.957259 4949 raft_consensus.cc:729] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a401541cb0c45dab919bc3ced47f6df, State: Initialized, Role: FOLLOWER
I20260929 18:31:23.957700 4949 consensus_queue.cc:260] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.958350 4798 heartbeater.cc:503] Master 127.0.98.62:33889 was elected leader, sending a full tablet report...
I20260929 18:31:23.958606 4949 ts_tablet_manager.cc:1428] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:31:23.975557 4666 tablet.cc:2369] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:23.975857 4954 raft_consensus.cc:482] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:23.976076 4954 raft_consensus.cc:504] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.977212 4954 leader_election.cc:288] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0804c8b116344914b650c9d43610b705 (127.0.98.1:32899), 2a401541cb0c45dab919bc3ced47f6df (127.0.98.2:42465)
I20260929 18:31:23.982055 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.977639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34339 (local address 127.0.98.1:32899)
0929 18:31:23.977775 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.977780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.978037 (+ 257us) server_negotiation.cc:407] Connection header received
0929 18:31:23.978139 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.978144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.978189 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.978255 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:23.978540 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.979253 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.980811 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.981433 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.981549 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.981608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.981855 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.981939 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:31:23.982055 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.977680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53917 (local address 127.0.98.2:42465)
0929 18:31:23.977809 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:23.977814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:23.977828 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:23.977910 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:23.977913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:23.977957 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:23.978049 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:23.978379 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.979151 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.980810 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.981433 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.981546 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.981608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:23.981855 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:31:23.981925 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:31:23.982087 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.977478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42465 (local address 127.0.98.3:53917)
0929 18:31:23.977699 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.977715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.977804 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.978082 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.978093 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.978111 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:23.978252 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.978261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.979268 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.979271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.980681 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.980691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.981569 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.981573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.981703 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.981744 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.981825 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.981928 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":61}
I20260929 18:31:23.982100 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:23.977504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32899 (local address 127.0.98.3:34339)
0929 18:31:23.977935 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:23.977946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:23.978051 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:23.978290 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:23.978296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:23.978317 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:23.978428 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.978433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.979353 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.979356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.980680 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:23.980691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:23.981569 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:23.981573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:23.981699 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:23.981756 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 18:31:23.981825 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:31:23.981932 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":90,"thread_start_us":144,"threads_started":1}
I20260929 18:31:23.982739 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c60efd9674678bd68c2ec19f24cd5" candidate_uuid: "afa050f254f44c099a8170f3cfc4145d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a401541cb0c45dab919bc3ced47f6df" is_pre_election: true
I20260929 18:31:23.982751 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c60efd9674678bd68c2ec19f24cd5" candidate_uuid: "afa050f254f44c099a8170f3cfc4145d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0804c8b116344914b650c9d43610b705" is_pre_election: true
I20260929 18:31:23.983052 4752 raft_consensus.cc:2454] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa050f254f44c099a8170f3cfc4145d in term 0.
I20260929 18:31:23.983055 4619 raft_consensus.cc:2454] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa050f254f44c099a8170f3cfc4145d in term 0.
I20260929 18:31:23.983577 4820 leader_election.cc:302] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0804c8b116344914b650c9d43610b705, 2a401541cb0c45dab919bc3ced47f6df, afa050f254f44c099a8170f3cfc4145d; no voters:
I20260929 18:31:23.983877 4954 raft_consensus.cc:2781] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:23.983984 4954 raft_consensus.cc:482] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:23.984027 4954 raft_consensus.cc:3037] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:23.985051 4954 raft_consensus.cc:504] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.985383 4954 leader_election.cc:288] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [CANDIDATE]: Term 1 election: Requested vote from peers 0804c8b116344914b650c9d43610b705 (127.0.98.1:32899), 2a401541cb0c45dab919bc3ced47f6df (127.0.98.2:42465)
I20260929 18:31:23.985736 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c60efd9674678bd68c2ec19f24cd5" candidate_uuid: "afa050f254f44c099a8170f3cfc4145d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0804c8b116344914b650c9d43610b705"
I20260929 18:31:23.985803 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c60efd9674678bd68c2ec19f24cd5" candidate_uuid: "afa050f254f44c099a8170f3cfc4145d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a401541cb0c45dab919bc3ced47f6df"
I20260929 18:31:23.985847 4619 raft_consensus.cc:3037] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:23.985946 4752 raft_consensus.cc:3037] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:23.986987 4619 raft_consensus.cc:2454] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa050f254f44c099a8170f3cfc4145d in term 1.
I20260929 18:31:23.986970 4752 raft_consensus.cc:2454] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa050f254f44c099a8170f3cfc4145d in term 1.
I20260929 18:31:23.987345 4818 leader_election.cc:302] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a401541cb0c45dab919bc3ced47f6df, afa050f254f44c099a8170f3cfc4145d; no voters:
I20260929 18:31:23.987507 4954 raft_consensus.cc:2781] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:23.987774 4954 raft_consensus.cc:686] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 1 LEADER]: Becoming Leader. State: Replica: afa050f254f44c099a8170f3cfc4145d, State: Running, Role: LEADER
I20260929 18:31:23.988135 4954 consensus_queue.cc:237] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } }
I20260929 18:31:23.991410 4474 catalog_manager.cc:5495] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d reported cstate change: term changed from 0 to 1, leader changed from <none> to afa050f254f44c099a8170f3cfc4145d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "afa050f254f44c099a8170f3cfc4145d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afa050f254f44c099a8170f3cfc4145d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44149 } health_report { overall_health: HEALTHY } } }
I20260929 18:31:24.011994 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.008311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.0.98.3:44149)
0929 18:31:24.008440 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.008447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.008465 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:31:24.008664 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.008667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.008725 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.008852 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:24.009113 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:24.009633 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:24.010500 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:24.010718 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:24.010825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:24.011148 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:24.011575 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:24.011717 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.011769 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.011823 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:31:24.017602 4752 raft_consensus.cc:1261] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 1 FOLLOWER]: Refusing update from remote peer afa050f254f44c099a8170f3cfc4145d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:24.017603 4619 raft_consensus.cc:1261] T 301c60efd9674678bd68c2ec19f24cd5 P 0804c8b116344914b650c9d43610b705 [term 1 FOLLOWER]: Refusing update from remote peer afa050f254f44c099a8170f3cfc4145d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:24.018276 4958 consensus_queue.cc:1035] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a401541cb0c45dab919bc3ced47f6df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:24.018474 4954 consensus_queue.cc:1035] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0804c8b116344914b650c9d43610b705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:24.026340 4963 mvcc.cc:205] Tried to move back new op lower bound from 7334734577729339392 to 7334734577618235392. Current Snapshot: MvccSnapshot[applied={T|T < 7334734577729339392}]
I20260929 18:31:24.026552 4962 mvcc.cc:205] Tried to move back new op lower bound from 7334734577729339392 to 7334734577618235392. Current Snapshot: MvccSnapshot[applied={T|T < 7334734577729339392}]
I20260929 18:31:24.035441 4474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:24.035586 4474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:24.038003 4474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29d7f71ab88a4449ab9f572c1a7da841: Sending DeleteTablet for 3 replicas of tablet 301c60efd9674678bd68c2ec19f24cd5
I20260929 18:31:24.038758 4599 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44534
I20260929 18:31:24.039047 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33422
I20260929 18:31:24.039248 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c60efd9674678bd68c2ec19f24cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50948
I20260929 18:31:24.039392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4537
I20260929 18:31:24.040343 4972 tablet_replica.cc:330] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: stopping tablet replica
I20260929 18:31:24.040352 4971 tablet_replica.cc:330] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: stopping tablet replica
I20260929 18:31:24.040660 4971 raft_consensus.cc:2229] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:24.040843 4972 raft_consensus.cc:2229] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:24.041078 4971 raft_consensus.cc:2258] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:24.041257 4972 raft_consensus.cc:2258] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:24.042634 4971 ts_tablet_manager.cc:1905] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:24.042682 4972 ts_tablet_manager.cc:1905] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:24.045430 4972 ts_tablet_manager.cc:1918] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:24.045583 4972 log.cc:1196] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/301c60efd9674678bd68c2ec19f24cd5
I20260929 18:31:24.045974 4972 ts_tablet_manager.cc:1939] T 301c60efd9674678bd68c2ec19f24cd5 P 2a401541cb0c45dab919bc3ced47f6df: Deleting consensus metadata
I20260929 18:31:24.046072 4971 ts_tablet_manager.cc:1918] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:24.046173 4971 log.cc:1196] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/301c60efd9674678bd68c2ec19f24cd5
I20260929 18:31:24.046525 4971 ts_tablet_manager.cc:1939] T 301c60efd9674678bd68c2ec19f24cd5 P afa050f254f44c099a8170f3cfc4145d: Deleting consensus metadata
I20260929 18:31:24.047463 4459 catalog_manager.cc:4841] TS 2a401541cb0c45dab919bc3ced47f6df (127.0.98.2:42465): tablet 301c60efd9674678bd68c2ec19f24cd5 (table test-table [id=cdbf00920c914d55975ce03247c9e1bd]) successfully deleted
I20260929 18:31:24.047700 4460 catalog_manager.cc:4841] TS afa050f254f44c099a8170f3cfc4145d (127.0.98.3:44149): tablet 301c60efd9674678bd68c2ec19f24cd5 (table test-table [id=cdbf00920c914d55975ce03247c9e1bd]) successfully deleted
W20260929 18:31:24.048746 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32899 (error 108)
I20260929 18:31:24.049605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4670
I20260929 18:31:24.049834 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.049495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32899 (local address 127.0.0.1:44548)
0929 18:31:24.049641 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.049724 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 18:31:24.051368 4461 catalog_manager.cc:4584] TS 0804c8b116344914b650c9d43610b705 (127.0.98.1:32899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 301c60efd9674678bd68c2ec19f24cd5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32899: connect: Connection refused (error 111)
I20260929 18:31:24.055945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4803
I20260929 18:31:24.061385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4444
2026-09-29T18:31:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2195 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:24Z Disabled control of system clock
I20260929 18:31:24.091045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45789
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:31:24.198025 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:24.198285 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:24.198321 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:24.202332 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:24.202405 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:24.202421 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:24.202437 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:24.202452 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:24.207274 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:24.208377 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:24.209754 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:24.218134 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.218099 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.218273 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:24.218400 4977 server_base.cc:1034] running on GCE node
I20260929 18:31:24.219131 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:24.220329 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:24.221522 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790706684221504 us; error 32 us; skew 500 ppm
I20260929 18:31:24.223278 4977 webserver.cc:460] Webserver started at http://127.0.98.62:35799/ using document root <none> and password file <none>
I20260929 18:31:24.223881 4977 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:24.223973 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:24.224263 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:24.226240 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "e53b62adafdf4fdbba07f9dab4ec1775"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "258173f2221f8971956cd1fc7bcd452a"
server_key_iv: "aecba6467dbb0cd11336dd5246168968"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.226794 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "e53b62adafdf4fdbba07f9dab4ec1775"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "258173f2221f8971956cd1fc7bcd452a"
server_key_iv: "aecba6467dbb0cd11336dd5246168968"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.230311 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:24.232709 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.233909 4977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:24.234046 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "e53b62adafdf4fdbba07f9dab4ec1775"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "258173f2221f8971956cd1fc7bcd452a"
server_key_iv: "aecba6467dbb0cd11336dd5246168968"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.234154 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:24.238582 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:24.239141 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:24.239317 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:24.245843 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45789
I20260929 18:31:24.245847 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45789 every 8 connection(s)
I20260929 18:31:24.246773 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:24.249799 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:24.255098 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: Bootstrap starting.
I20260929 18:31:24.256462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 4977
I20260929 18:31:24.256558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:24.256867 392 external_mini_cluster.cc:1428] Setting key 0fab59d80835a35bbf46fbd651e76f00
I20260929 18:31:24.257778 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:24.258486 5044 log.cc:824] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:24.260442 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: No bootstrap required, opened a new log
I20260929 18:31:24.262244 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.258016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.98.62:45789)
0929 18:31:24.258748 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.258783 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.258819 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:31:24.259480 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.259498 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.259802 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.260084 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.260935 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.260940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.260951 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.260968 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.261020 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.261036 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.261134 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.261274 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.261549 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":169,"threads_started":1}
I20260929 18:31:24.264369 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260929 18:31:24.264633 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:24.264712 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53b62adafdf4fdbba07f9dab4ec1775, State: Initialized, Role: FOLLOWER
I20260929 18:31:24.265151 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260929 18:31:24.265296 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:24.265363 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:24.265457 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:24.266302 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260929 18:31:24.266629 5044 leader_election.cc:302] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53b62adafdf4fdbba07f9dab4ec1775; no voters:
I20260929 18:31:24.266922 5044 leader_election.cc:288] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:24.267081 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:24.267378 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [term 1 LEADER]: Becoming Leader. State: Replica: e53b62adafdf4fdbba07f9dab4ec1775, State: Running, Role: LEADER
I20260929 18:31:24.267660 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260929 18:31:24.268108 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:24.269464 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } } }
I20260929 18:31:24.269440 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53b62adafdf4fdbba07f9dab4ec1775. Latest consensus state: current_term: 1 leader_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b62adafdf4fdbba07f9dab4ec1775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } } }
I20260929 18:31:24.269611 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:24.269611 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:24.270056 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:24.273432 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:24.278342 5060 catalog_manager.cc:1347] Generated new cluster ID: c580954165bc473d9b18ffc1e9a83214
I20260929 18:31:24.278411 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:24.295179 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:24.295497 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:24.303853 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: Generated new TSK 0
I20260929 18:31:24.304550 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:24.316484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--builtin_ntp_servers=127.0.98.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:31:24.427049 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:24.427301 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:24.427340 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:24.431353 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:24.431407 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:24.431484 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:24.436326 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:24.437336 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:24.438715 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:24.448004 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.448004 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.448027 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:24.448364 5068 server_base.cc:1034] running on GCE node
I20260929 18:31:24.448743 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:24.449438 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:24.450695 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790706684450681 us; error 46 us; skew 500 ppm
I20260929 18:31:24.452669 5068 webserver.cc:460] Webserver started at http://127.0.98.1:36915/ using document root <none> and password file <none>
I20260929 18:31:24.453230 5068 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:24.453329 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:24.453567 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:24.455524 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "8dfa5d92e4eab6ca24179216ce0d0899"
server_key_iv: "cfd7e6dc60dc0d9072405fca25e4ff26"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.456079 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "8dfa5d92e4eab6ca24179216ce0d0899"
server_key_iv: "cfd7e6dc60dc0d9072405fca25e4ff26"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.459486 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:24.461956 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.463400 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:24.463557 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "8dfa5d92e4eab6ca24179216ce0d0899"
server_key_iv: "cfd7e6dc60dc0d9072405fca25e4ff26"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.463676 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:24.468636 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:24.469347 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:24.469553 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:24.470162 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:24.471108 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:24.471184 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.471268 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:24.471304 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.480695 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45097
I20260929 18:31:24.480723 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45097 every 8 connection(s)
I20260929 18:31:24.481611 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:24.482563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5068
I20260929 18:31:24.482708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:24.483062 392 external_mini_cluster.cc:1428] Setting key a7d077b8cec09ce00e3db83ce42722b3
I20260929 18:31:24.485361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--builtin_ntp_servers=127.0.98.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:31:24.489614 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.484366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41163 (local address 127.0.98.62:45789)
0929 18:31:24.484535 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.484542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.485802 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 18:31:24.487673 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.487676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.487719 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.487800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.489126 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.489128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.489130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.489141 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.489159 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.489167 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.489244 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.489460 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.489508 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:24.490924 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.484124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.1:41163)
0929 18:31:24.485633 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.485694 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.487448 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.487992 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.488004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.488131 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.488802 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.488807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.488866 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.488966 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.489264 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.489276 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.489286 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.489484 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.489756 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":149,"threads_started":1}
I20260929 18:31:24.492184 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260929 18:31:24.492411 5196 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:24.492798 5196 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260929 18:31:24.494485 5008 ts_manager.cc:194] Registered new tserver with Master: 4c2e6bb4eeec46daa18e8365f23cd9f1 (127.0.98.1:45097)
W20260929 18:31:24.594515 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:24.594744 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:24.594780 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:24.598670 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:24.598716 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:24.598767 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:24.603246 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:24.604597 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:24.606431 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:24.617075 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.617071 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.617072 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:24.617404 5199 server_base.cc:1034] running on GCE node
I20260929 18:31:24.617763 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:24.618378 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:24.619536 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790706684619518 us; error 29 us; skew 500 ppm
I20260929 18:31:24.621243 5199 webserver.cc:460] Webserver started at http://127.0.98.2:46415/ using document root <none> and password file <none>
I20260929 18:31:24.621771 5199 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:24.621829 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:24.621981 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:24.623965 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b254aeb530a4231a74f5c24c6f9165e"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "cdbc7a791d88c5b5189d6c5fdcd6c849"
server_key_iv: "1c0e3aca8f0906bed01cc9272ec4f347"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.624552 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b254aeb530a4231a74f5c24c6f9165e"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "cdbc7a791d88c5b5189d6c5fdcd6c849"
server_key_iv: "1c0e3aca8f0906bed01cc9272ec4f347"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.628073 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:24.630424 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.631595 5199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:24.631685 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "9b254aeb530a4231a74f5c24c6f9165e"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "cdbc7a791d88c5b5189d6c5fdcd6c849"
server_key_iv: "1c0e3aca8f0906bed01cc9272ec4f347"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.631762 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:24.635968 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:24.636564 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:24.636695 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:24.637214 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:24.638114 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:24.638160 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.638219 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:24.638254 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.646503 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43623
I20260929 18:31:24.646523 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43623 every 8 connection(s)
I20260929 18:31:24.647308 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:24.651887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5199
I20260929 18:31:24.652055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:24.652731 392 external_mini_cluster.cc:1428] Setting key e796505337a2ef9f32b74675f6fce263
I20260929 18:31:24.653286 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.649049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48225 (local address 127.0.98.62:45789)
0929 18:31:24.649177 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.649181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.650190 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 18:31:24.651476 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.651480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.651524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.651635 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.652830 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.652833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.652836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.652849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.652873 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.652883 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.652971 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.653112 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.653156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:24.654211 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.649428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.2:48225)
0929 18:31:24.650048 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.650086 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.651261 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.651799 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.651806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.651900 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.652581 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.652582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.652621 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.652685 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.652954 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.652961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.652966 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.653143 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.653391 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260929 18:31:24.654913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--builtin_ntp_servers=127.0.98.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:31:24.655346 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260929 18:31:24.655575 5327 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:24.656030 5327 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260929 18:31:24.657107 5008 ts_manager.cc:194] Registered new tserver with Master: 9b254aeb530a4231a74f5c24c6f9165e (127.0.98.2:43623)
W20260929 18:31:24.762794 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:24.763036 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:24.763073 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:24.767009 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:24.767112 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:24.767198 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:24.772364 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:24.773384 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:24.774787 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:24.784132 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.784119 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:24.784123 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:24.784193 5330 server_base.cc:1034] running on GCE node
I20260929 18:31:24.784775 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:24.785384 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:24.786572 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790706684786559 us; error 27 us; skew 500 ppm
I20260929 18:31:24.788417 5330 webserver.cc:460] Webserver started at http://127.0.98.3:36987/ using document root <none> and password file <none>
I20260929 18:31:24.788968 5330 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:24.789058 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:24.789280 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:24.791442 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3645fd78dfb4a689a3b4c35c85a1b87"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "01f9acc403fab940d7882669d0abdecb"
server_key_iv: "fc2140953ba270eedfcb18922de22083"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.791991 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3645fd78dfb4a689a3b4c35c85a1b87"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "01f9acc403fab940d7882669d0abdecb"
server_key_iv: "fc2140953ba270eedfcb18922de22083"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.795459 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:24.797825 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.798934 5330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:24.799062 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "d3645fd78dfb4a689a3b4c35c85a1b87"
format_stamp: "Formatted at 2026-09-29 18:31:24 on dist-test-slave-fmv8"
server_key: "01f9acc403fab940d7882669d0abdecb"
server_key_iv: "fc2140953ba270eedfcb18922de22083"
server_key_version: "encryptionkey@0"
I20260929 18:31:24.799166 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:24.803608 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:24.804229 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:24.804395 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:24.804914 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:24.805752 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:24.805821 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.805914 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:24.805964 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:24.814617 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41987
I20260929 18:31:24.814668 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41987 every 8 connection(s)
I20260929 18:31:24.815490 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:24.820969 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.817264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34417 (local address 127.0.98.62:45789)
0929 18:31:24.817397 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.817401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.818348 (+ 947us) server_negotiation.cc:407] Connection header received
0929 18:31:24.819534 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.819537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.819571 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.819644 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.820623 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.820626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.820628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.820638 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.820657 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.820665 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.820742 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.820832 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.820866 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:31:24.821363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5330
I20260929 18:31:24.821485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:24.821762 392 external_mini_cluster.cc:1428] Setting key 2bd386ee29d0936afda20c43fa81f4e1
I20260929 18:31:24.822005 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.817586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.3:34417)
0929 18:31:24.818199 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.818237 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.819358 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.819765 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.819771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.819861 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.820391 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.820393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.820429 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.820492 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.820729 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.820733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.820736 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.820853 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.821058 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260929 18:31:24.823096 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260929 18:31:24.823318 5458 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:24.823771 5458 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260929 18:31:24.824750 5007 ts_manager.cc:194] Registered new tserver with Master: d3645fd78dfb4a689a3b4c35c85a1b87 (127.0.98.3:41987)
I20260929 18:31:24.835749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:31:24.839309 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.837861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.0.98.62:45789)
0929 18:31:24.837989 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.837996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.838012 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:31:24.838053 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.838060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.838089 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.838171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.838895 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.838899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.838902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.838913 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.838932 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.838948 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.839033 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.839116 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.839158 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:31:24.842417 5007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:24.845472 5007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:24.856494 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.853515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45097 (local address 127.0.0.1:35608)
0929 18:31:24.854145 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.854190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.854286 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.855032 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.855038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.855115 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.855660 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.855661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.855676 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.855729 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.856139 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.856168 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.856174 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.856285 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.856353 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":80,"threads_started":1}
I20260929 18:31:24.856525 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.853663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.0.98.1:45097)
0929 18:31:24.854437 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.854453 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.854481 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:31:24.854534 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.854539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.854730 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.854888 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.855898 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.855907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.855922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.855939 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.855990 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.856030 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.856157 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.856302 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.856372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":138,"threads_started":1}
I20260929 18:31:24.857290 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.853671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43623 (local address 127.0.0.1:38376)
0929 18:31:24.854857 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.854870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.855003 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.856140 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.856167 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.856186 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.856601 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.856603 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.856615 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.856650 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.857069 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.857072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.857075 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.857144 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.857176 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":99,"threads_started":1}
I20260929 18:31:24.857738 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.854278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.0.98.2:43623)
0929 18:31:24.854710 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.854721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.854946 (+ 225us) server_negotiation.cc:407] Connection header received
0929 18:31:24.855128 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.855136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.855365 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.855480 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.856797 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.856804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.856815 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.856827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.856865 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.856912 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.857014 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.857519 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.857585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":113,"threads_started":1}
I20260929 18:31:24.858608 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.853972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41987 (local address 127.0.0.1:55670)
0929 18:31:24.854996 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.855008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.855074 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.855841 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.855844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.855860 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.857010 (+ 1150us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.857012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.857021 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.857045 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.858359 (+ 1314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.858363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.858368 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.858460 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.858497 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":83,"threads_started":1}
I20260929 18:31:24.858695 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.854318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.98.3:41987)
0929 18:31:24.854725 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.854736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.855165 (+ 429us) server_negotiation.cc:407] Connection header received
0929 18:31:24.855221 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.855225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.855693 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.855848 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.858173 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.858179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.858188 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.858198 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.858233 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.858254 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.858367 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.858483 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.858545 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":130,"threads_started":1}
I20260929 18:31:24.859622 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 (DEFAULT_TABLE table=test-table [id=9be993f2fc094bd9b4bf5116eac5f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:24.860278 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 (DEFAULT_TABLE table=test-table [id=9be993f2fc094bd9b4bf5116eac5f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:24.860771 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f17f48dfdfa4505b79f8061da4e0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:24.861168 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f17f48dfdfa4505b79f8061da4e0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:24.861408 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 (DEFAULT_TABLE table=test-table [id=9be993f2fc094bd9b4bf5116eac5f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:24.862283 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f17f48dfdfa4505b79f8061da4e0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:24.867272 5475 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Bootstrap starting.
I20260929 18:31:24.869527 5477 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Bootstrap starting.
I20260929 18:31:24.870051 5476 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: Bootstrap starting.
I20260929 18:31:24.870240 5475 tablet_bootstrap.cc:654] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:24.870954 5475 log.cc:824] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:24.871704 5477 tablet_bootstrap.cc:654] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:24.871984 5476 tablet_bootstrap.cc:654] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:24.872527 5477 log.cc:824] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:24.872718 5476 log.cc:824] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:24.872937 5475 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: No bootstrap required, opened a new log
I20260929 18:31:24.873126 5475 ts_tablet_manager.cc:1397] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:31:24.874547 5477 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: No bootstrap required, opened a new log
I20260929 18:31:24.874547 5476 tablet_bootstrap.cc:492] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: No bootstrap required, opened a new log
I20260929 18:31:24.874715 5477 ts_tablet_manager.cc:1397] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:31:24.874715 5476 ts_tablet_manager.cc:1397] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 18:31:24.876657 5475 raft_consensus.cc:348] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.876895 5475 raft_consensus.cc:374] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:24.876946 5475 raft_consensus.cc:729] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b254aeb530a4231a74f5c24c6f9165e, State: Initialized, Role: FOLLOWER
I20260929 18:31:24.877380 5475 consensus_queue.cc:260] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.877995 5327 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
I20260929 18:31:24.878350 5475 ts_tablet_manager.cc:1428] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:24.879122 5476 raft_consensus.cc:348] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.879122 5477 raft_consensus.cc:348] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.879364 5477 raft_consensus.cc:374] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:24.879364 5476 raft_consensus.cc:374] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:24.879412 5476 raft_consensus.cc:729] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2e6bb4eeec46daa18e8365f23cd9f1, State: Initialized, Role: FOLLOWER
I20260929 18:31:24.879412 5477 raft_consensus.cc:729] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3645fd78dfb4a689a3b4c35c85a1b87, State: Initialized, Role: FOLLOWER
I20260929 18:31:24.879853 5477 consensus_queue.cc:260] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.879853 5476 consensus_queue.cc:260] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.880771 5476 ts_tablet_manager.cc:1428] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 18:31:24.880771 5477 ts_tablet_manager.cc:1428] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:31:24.881026 5196 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
I20260929 18:31:24.881258 5458 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
W20260929 18:31:24.898201 5328 tablet.cc:2369] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:24.917120 5482 raft_consensus.cc:482] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:24.917447 5482 raft_consensus.cc:504] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.918638 5482 leader_election.cc:288] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b254aeb530a4231a74f5c24c6f9165e (127.0.98.2:43623), d3645fd78dfb4a689a3b4c35c85a1b87 (127.0.98.3:41987)
I20260929 18:31:24.921123 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.918952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41987 (local address 127.0.98.1:44875)
0929 18:31:24.919084 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.919101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.919233 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.919486 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.919490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.919508 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.920065 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.920067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.920080 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.920106 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.920732 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.920734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.920737 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.920946 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.920986 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:31:24.921145 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.918971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43623 (local address 127.0.98.1:49989)
0929 18:31:24.919705 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.919717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:24.919789 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:24.920052 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:24.920055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:24.920071 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:24.920508 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:24.920509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:24.920518 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:24.920541 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:24.920902 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:24.920909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:24.920912 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:31:24.920986 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:31:24.921010 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":332,"thread_start_us":153,"threads_started":1}
I20260929 18:31:24.921195 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.919119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44875 (local address 127.0.98.3:41987)
0929 18:31:24.919265 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.919268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.919283 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:24.919320 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.919323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.919354 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.919423 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.920544 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.920547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.920549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.920561 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.920580 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.920614 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.920717 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.921023 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.921069 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:31:24.921195 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.919043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49989 (local address 127.0.98.2:43623)
0929 18:31:24.919189 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.919195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.919798 (+ 603us) server_negotiation.cc:407] Connection header received
0929 18:31:24.919849 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.919853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.919896 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.919984 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.920664 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.920671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.920674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.920689 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.920709 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.920718 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.920801 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.921025 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.921069 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:31:24.921861 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f17f48dfdfa4505b79f8061da4e0669" candidate_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" is_pre_election: true
I20260929 18:31:24.921862 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f17f48dfdfa4505b79f8061da4e0669" candidate_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b254aeb530a4231a74f5c24c6f9165e" is_pre_election: true
I20260929 18:31:24.922142 5412 raft_consensus.cc:2454] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2e6bb4eeec46daa18e8365f23cd9f1 in term 0.
I20260929 18:31:24.922165 5281 raft_consensus.cc:2454] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2e6bb4eeec46daa18e8365f23cd9f1 in term 0.
I20260929 18:31:24.922566 5086 leader_election.cc:302] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2e6bb4eeec46daa18e8365f23cd9f1, 9b254aeb530a4231a74f5c24c6f9165e; no voters:
I20260929 18:31:24.922851 5482 raft_consensus.cc:2781] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:24.922950 5482 raft_consensus.cc:482] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:24.923007 5482 raft_consensus.cc:3037] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:24.923991 5482 raft_consensus.cc:504] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.924376 5482 leader_election.cc:288] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [CANDIDATE]: Term 1 election: Requested vote from peers 9b254aeb530a4231a74f5c24c6f9165e (127.0.98.2:43623), d3645fd78dfb4a689a3b4c35c85a1b87 (127.0.98.3:41987)
I20260929 18:31:24.924636 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f17f48dfdfa4505b79f8061da4e0669" candidate_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b254aeb530a4231a74f5c24c6f9165e"
I20260929 18:31:24.924675 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f17f48dfdfa4505b79f8061da4e0669" candidate_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87"
I20260929 18:31:24.924746 5281 raft_consensus.cc:3037] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:24.924826 5412 raft_consensus.cc:3037] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:24.925740 5281 raft_consensus.cc:2454] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2e6bb4eeec46daa18e8365f23cd9f1 in term 1.
I20260929 18:31:24.925935 5412 raft_consensus.cc:2454] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2e6bb4eeec46daa18e8365f23cd9f1 in term 1.
I20260929 18:31:24.926038 5086 leader_election.cc:302] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2e6bb4eeec46daa18e8365f23cd9f1, 9b254aeb530a4231a74f5c24c6f9165e; no voters:
I20260929 18:31:24.926220 5482 raft_consensus.cc:2781] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:24.926419 5482 raft_consensus.cc:686] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2e6bb4eeec46daa18e8365f23cd9f1, State: Running, Role: LEADER
I20260929 18:31:24.926720 5482 consensus_queue.cc:237] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } }
I20260929 18:31:24.929656 5005 catalog_manager.cc:5495] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c2e6bb4eeec46daa18e8365f23cd9f1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2e6bb4eeec46daa18e8365f23cd9f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:24.939139 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.937509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.0.98.1:45097)
0929 18:31:24.937618 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:24.937622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:24.937665 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:31:24.937862 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:24.937865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:24.937915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:24.938011 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:24.938753 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:24.938757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:24.938759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:24.938770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:24.938790 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:24.938798 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:24.938899 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:24.938978 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:31:24.939016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:31:24.945477 5281 raft_consensus.cc:1261] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 1 FOLLOWER]: Refusing update from remote peer 4c2e6bb4eeec46daa18e8365f23cd9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:24.945609 5412 raft_consensus.cc:1261] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 1 FOLLOWER]: Refusing update from remote peer 4c2e6bb4eeec46daa18e8365f23cd9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:24.946220 5482 consensus_queue.cc:1035] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b254aeb530a4231a74f5c24c6f9165e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:24.946393 5485 consensus_queue.cc:1035] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3645fd78dfb4a689a3b4c35c85a1b87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:24.957221 5489 mvcc.cc:205] Tried to move back new op lower bound from 7334734581528526848 to 7334734581462372352. Current Snapshot: MvccSnapshot[applied={T|T < 7334734581528526848 or (T in {7334734581528526848})}]
I20260929 18:31:24.961822 5005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:24.961962 5005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:24.964591 5005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e53b62adafdf4fdbba07f9dab4ec1775: Sending DeleteTablet for 3 replicas of tablet 7f17f48dfdfa4505b79f8061da4e0669
I20260929 18:31:24.965350 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='slave'} at 127.0.0.1:38376
I20260929 18:31:24.965351 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='slave'} at 127.0.0.1:35608
I20260929 18:31:24.965803 5497 tablet_replica.cc:330] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1: stopping tablet replica
I20260929 18:31:24.965830 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f17f48dfdfa4505b79f8061da4e0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:24 UTC) from {username='slave'} at 127.0.0.1:55670
I20260929 18:31:24.966238 5497 raft_consensus.cc:2229] T 7f17f48dfdfa4505b79f8061da4e0669 P 4c2e6bb4eeec46daa18e8365f23cd9f1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:24.966286 5499 tablet_replica.cc:330] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: stopping tablet replica
I20260929 18:31:24.966434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5068
I20260929 18:31:24.967079 5498 tablet_replica.cc:330] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: stopping tablet replica
I20260929 18:31:24.967379 5498 raft_consensus.cc:2229] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:24.967674 5498 raft_consensus.cc:2258] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:24.966585 5499 raft_consensus.cc:2229] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:24.968139 5499 raft_consensus.cc:2258] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:24.969571 5498 ts_tablet_manager.cc:1905] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:24.971696 5498 ts_tablet_manager.cc:1918] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:24.971774 5498 log.cc:1196] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/7f17f48dfdfa4505b79f8061da4e0669
I20260929 18:31:24.972064 5498 ts_tablet_manager.cc:1939] T 7f17f48dfdfa4505b79f8061da4e0669 P 9b254aeb530a4231a74f5c24c6f9165e: Deleting consensus metadata
I20260929 18:31:24.973104 4995 catalog_manager.cc:4841] TS 9b254aeb530a4231a74f5c24c6f9165e (127.0.98.2:43623): tablet 7f17f48dfdfa4505b79f8061da4e0669 (table test-table [id=9be993f2fc094bd9b4bf5116eac5f3ef]) successfully deleted
W20260929 18:31:24.973987 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45097 (error 108)
I20260929 18:31:24.974026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5199
I20260929 18:31:24.975158 5499 ts_tablet_manager.cc:1905] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:24.976186 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:24.975870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45097 (local address 127.0.0.1:35638)
0929 18:31:24.976008 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:24.976080 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:31:24.976375 4994 catalog_manager.cc:4584] TS 4c2e6bb4eeec46daa18e8365f23cd9f1 (127.0.98.1:45097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f17f48dfdfa4505b79f8061da4e0669: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45097: connect: Connection refused (error 111)
I20260929 18:31:24.977429 5499 ts_tablet_manager.cc:1918] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:24.977537 5499 log.cc:1196] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/7f17f48dfdfa4505b79f8061da4e0669
I20260929 18:31:24.977813 5499 ts_tablet_manager.cc:1939] T 7f17f48dfdfa4505b79f8061da4e0669 P d3645fd78dfb4a689a3b4c35c85a1b87: Deleting consensus metadata
I20260929 18:31:24.978852 4993 catalog_manager.cc:4841] TS d3645fd78dfb4a689a3b4c35c85a1b87 (127.0.98.3:41987): tablet 7f17f48dfdfa4505b79f8061da4e0669 (table test-table [id=9be993f2fc094bd9b4bf5116eac5f3ef]) successfully deleted
I20260929 18:31:24.980430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5330
I20260929 18:31:24.986096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 4977
2026-09-29T18:31:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (925 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:31:25.009168 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:31:25.010738 392 webserver.cc:460] Webserver started at https://127.0.0.1:36499/ using document root <none> and password file <none>
I20260929 18:31:25.010911 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:31:25.011206 392 webserver.cc:460] Webserver started at http://127.0.0.1:40377/ using document root <none> and password file <none>
2026-09-29T18:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:25Z Disabled control of system clock
I20260929 18:31:25.020190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35827
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36499/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/testchainca.cert with env {}
W20260929 18:31:25.128039 5506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:25.128320 5506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:25.128360 5506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:25.132246 5506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:25.132297 5506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:25.132313 5506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:25.132329 5506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:25.132344 5506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:31:25.132359 5506 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:31:25.132372 5506 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36499/jwks.json
I20260929 18:31:25.137586 5506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36499/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:25.138526 5506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:25.139428 5506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:25.147570 5511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.147568 5512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.148010 5514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:25.148003 5506 server_base.cc:1034] running on GCE node
I20260929 18:31:25.148571 5506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:25.149601 5506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:25.150768 5506 hybrid_clock.cc:648] HybridClock initialized: now 1790706685150757 us; error 25 us; skew 500 ppm
I20260929 18:31:25.152704 5506 webserver.cc:460] Webserver started at http://127.0.98.62:33639/ using document root <none> and password file <none>
I20260929 18:31:25.153262 5506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:25.153323 5506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:25.153548 5506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:25.155505 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "b431ee25e5ad4eccadb5d4420214c757"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "ea103e3faf9955bb24605e14860ec7ff"
server_key_iv: "2615da524edb7817dc474267d825d394"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.156070 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "b431ee25e5ad4eccadb5d4420214c757"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "ea103e3faf9955bb24605e14860ec7ff"
server_key_iv: "2615da524edb7817dc474267d825d394"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.159533 5506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:25.161914 5520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:25.163084 5506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:31:25.163180 5506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "b431ee25e5ad4eccadb5d4420214c757"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "ea103e3faf9955bb24605e14860ec7ff"
server_key_iv: "2615da524edb7817dc474267d825d394"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.163306 5506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:25.182991 5506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:25.183696 5506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:25.184031 5506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:25.191114 5506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35827
I20260929 18:31:25.191124 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35827 every 8 connection(s)
I20260929 18:31:25.192214 5506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:25.195345 5573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:25.195994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5506
I20260929 18:31:25.196127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:25.196476 392 external_mini_cluster.cc:1428] Setting key c03a141585b37f910e4a743eac24edd5
I20260929 18:31:25.201542 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757: Bootstrap starting.
I20260929 18:31:25.203930 5573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:25.204720 5573 log.cc:824] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:25.206075 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.197871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.98.62:35827)
0929 18:31:25.198574 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.198588 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.198623 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:31:25.199316 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.199332 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.199720 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.200259 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:25.200710 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.201663 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.202638 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.203116 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.203290 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.204186 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:25.204194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:25.204210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:25.204599 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:25.204650 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:25.204669 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:25.204781 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:25.204935 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 18:31:25.205212 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":126,"threads_started":1}
I20260929 18:31:25.206851 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757: No bootstrap required, opened a new log
I20260929 18:31:25.208853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:31:25.210816 5573 raft_consensus.cc:348] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } }
I20260929 18:31:25.211040 5573 raft_consensus.cc:374] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:25.211114 5573 raft_consensus.cc:729] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b431ee25e5ad4eccadb5d4420214c757, State: Initialized, Role: FOLLOWER
I20260929 18:31:25.211576 5573 consensus_queue.cc:260] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } }
I20260929 18:31:25.211726 5573 raft_consensus.cc:388] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:25.211791 5573 raft_consensus.cc:482] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:25.211906 5573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:25.212847 5573 raft_consensus.cc:504] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } }
I20260929 18:31:25.213184 5573 leader_election.cc:302] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b431ee25e5ad4eccadb5d4420214c757; no voters:
I20260929 18:31:25.213483 5573 leader_election.cc:288] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:25.213618 5578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:25.214097 5578 raft_consensus.cc:686] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [term 1 LEADER]: Becoming Leader. State: Replica: b431ee25e5ad4eccadb5d4420214c757, State: Running, Role: LEADER
I20260929 18:31:25.214411 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } }
I20260929 18:31:25.214793 5573 sys_catalog.cc:564] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:25.216089 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b431ee25e5ad4eccadb5d4420214c757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } } }
I20260929 18:31:25.216313 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.216087 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b431ee25e5ad4eccadb5d4420214c757. Latest consensus state: current_term: 1 leader_uuid: "b431ee25e5ad4eccadb5d4420214c757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431ee25e5ad4eccadb5d4420214c757" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35827 } } }
I20260929 18:31:25.216395 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.216827 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:25.219710 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:25.224872 5591 catalog_manager.cc:1347] Generated new cluster ID: 02ef9c4c1e7e4ab6ae2bc11e5bf262e5
I20260929 18:31:25.224931 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:25.234884 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:25.235993 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:25.242223 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b431ee25e5ad4eccadb5d4420214c757: Generated new TSK 0
I20260929 18:31:25.242920 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:31:25.273448 5603 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:31:25.273561 5603 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:31:25.274143 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.257700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.0.98.62:35827)
0929 18:31:25.257849 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.257854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.258061 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:31:25.258351 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.258354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.258403 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.258512 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:31:25.258785 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.259315 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.260240 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.260461 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.260573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.260907 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:31:25.273739 (+ 12832us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:31:25.274001 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 18:31:25.274211 5576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 18:31:25.275250 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:25.257609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35827 (local address 127.0.0.1:49982)
0929 18:31:25.257968 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:25.257980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:25.258127 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:25.258527 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:25.258531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:25.258540 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:31:25.258670 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.258676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.259428 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.259435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.260073 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.260080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.260563 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.260581 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.260675 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.260725 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:31:25.273934 (+ 13209us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:31:25.274138 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:31:25.275920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5506
2026-09-29T18:31:25Z chronyd exiting
I20260929 18:31:25.312327 392 test_util.cc:175] -----------------------------------------------
I20260929 18:31:25.312429 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706655775522-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:31:25.313446 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:31:25.314292 392 webserver.cc:460] Webserver started at https://127.0.0.1:38537/ using document root <none> and password file <none>
I20260929 18:31:25.314421 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:31:25.314723 392 webserver.cc:460] Webserver started at http://127.0.0.1:43145/ using document root <none> and password file <none>
2026-09-29T18:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:25Z Disabled control of system clock
I20260929 18:31:25.323024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38537/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/testchainca.cert with env {}
W20260929 18:31:25.441756 5609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:25.442029 5609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:25.442109 5609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:25.446058 5609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:25.446134 5609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:25.446170 5609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:25.446226 5609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:25.446264 5609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:31:25.446297 5609 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:31:25.446328 5609 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38537/jwks.json
I20260929 18:31:25.451512 5609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38537/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:25.452528 5609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:25.453495 5609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:25.461529 5615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.461503 5614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.461534 5617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:25.461766 5609 server_base.cc:1034] running on GCE node
I20260929 18:31:25.462366 5609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:25.463395 5609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:25.464600 5609 hybrid_clock.cc:648] HybridClock initialized: now 1790706685464581 us; error 31 us; skew 500 ppm
I20260929 18:31:25.466337 5609 webserver.cc:460] Webserver started at http://127.0.98.62:41171/ using document root <none> and password file <none>
I20260929 18:31:25.466890 5609 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:25.466991 5609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:25.467211 5609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:25.469542 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "a983d1ca7086450ab6394eace8c29ccf"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "8ace0f7a26adc2c49e59b8dc8fc6daa9"
server_key_iv: "503927e096a57d6429625110c656cc38"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.470096 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "a983d1ca7086450ab6394eace8c29ccf"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "8ace0f7a26adc2c49e59b8dc8fc6daa9"
server_key_iv: "503927e096a57d6429625110c656cc38"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.473702 5609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:25.475934 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:25.477125 5609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:25.477284 5609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "a983d1ca7086450ab6394eace8c29ccf"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "8ace0f7a26adc2c49e59b8dc8fc6daa9"
server_key_iv: "503927e096a57d6429625110c656cc38"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.477393 5609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:25.492950 5609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:25.493577 5609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:25.493764 5609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:25.500281 5609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45303
I20260929 18:31:25.500293 5675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45303 every 8 connection(s)
I20260929 18:31:25.501138 5609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:25.504313 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:25.508541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5609
I20260929 18:31:25.508647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:25.508942 392 external_mini_cluster.cc:1428] Setting key a0e425500c87e8eeb47392f6a5ecf083
I20260929 18:31:25.509887 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf: Bootstrap starting.
I20260929 18:31:25.512269 5676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:25.513000 5676 log.cc:824] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:25.514763 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf: No bootstrap required, opened a new log
I20260929 18:31:25.518090 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.510315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.0.98.62:45303)
0929 18:31:25.510990 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.511004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.511039 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:31:25.511704 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.511720 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.512120 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.512545 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:25.512979 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.513965 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.514910 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.515405 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.515579 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.516462 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:25.516474 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:25.516487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:25.516854 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:25.516907 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:25.516925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:25.517033 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:25.517150 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:31:25.517399 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":137,"threads_started":1}
I20260929 18:31:25.518390 5676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260929 18:31:25.518620 5676 raft_consensus.cc:374] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:25.518705 5676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a983d1ca7086450ab6394eace8c29ccf, State: Initialized, Role: FOLLOWER
I20260929 18:31:25.519213 5676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260929 18:31:25.519376 5676 raft_consensus.cc:388] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:25.519440 5676 raft_consensus.cc:482] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:25.519531 5676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:25.520684 5676 raft_consensus.cc:504] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260929 18:31:25.521037 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:31:25.521030 5676 leader_election.cc:302] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a983d1ca7086450ab6394eace8c29ccf; no voters:
I20260929 18:31:25.521349 5676 leader_election.cc:288] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:25.521440 5681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:25.521639 5681 raft_consensus.cc:686] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [term 1 LEADER]: Becoming Leader. State: Replica: a983d1ca7086450ab6394eace8c29ccf, State: Running, Role: LEADER
I20260929 18:31:25.521940 5681 consensus_queue.cc:237] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260929 18:31:25.522598 5676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:25.523087 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a983d1ca7086450ab6394eace8c29ccf. Latest consensus state: current_term: 1 leader_uuid: "a983d1ca7086450ab6394eace8c29ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } } }
I20260929 18:31:25.523214 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.523614 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a983d1ca7086450ab6394eace8c29ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a983d1ca7086450ab6394eace8c29ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } } }
I20260929 18:31:25.523712 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.523952 5686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:25.527052 5686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:25.533366 5686 catalog_manager.cc:1347] Generated new cluster ID: 753be6c533ae409da85f2a1dd072ab2e
I20260929 18:31:25.533452 5686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:25.549978 5686 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:25.551177 5686 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:25.563126 5686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a983d1ca7086450ab6394eace8c29ccf: Generated new TSK 0
I20260929 18:31:25.563822 5686 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:31:25.603488 5706 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:31:25.603581 5706 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:31:25.604295 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.581258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.0.98.62:45303)
0929 18:31:25.581378 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.581385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.581579 (+ 194us) server_negotiation.cc:407] Connection header received
0929 18:31:25.581772 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.581774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.581817 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.581922 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:31:25.582184 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.582720 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.583605 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.583832 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.583942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.584252 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:31:25.603821 (+ 19569us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:31:25.604107 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33988: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:31:25.604373 5679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33988: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:31:25.604655 5705 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:25.581169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45303 (local address 127.0.0.1:33988)
0929 18:31:25.581493 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:25.581505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:25.581630 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:25.581932 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:25.581936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:25.581944 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:31:25.582061 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.582068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.582835 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.582838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.583474 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.583481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.583932 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.583950 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.584035 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.584067 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:31:25.604022 (+ 19955us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:31:25.604217 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
I20260929 18:31:25.605195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5609
2026-09-29T18:31:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:31:25.644259 392 mini_oidc.cc:152] Starting JWKS server
I20260929 18:31:25.645350 392 webserver.cc:460] Webserver started at https://127.0.0.1:36203/ using document root <none> and password file <none>
I20260929 18:31:25.645438 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:31:25.645653 392 webserver.cc:460] Webserver started at http://127.0.0.1:36807/ using document root <none> and password file <none>
2026-09-29T18:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:25Z Disabled control of system clock
I20260929 18:31:25.654322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36817
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36203/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:31:25.761005 5712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:25.761255 5712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:25.761291 5712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:25.765223 5712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:25.765276 5712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:25.765292 5712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:25.765308 5712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:25.765324 5712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:31:25.765339 5712 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:31:25.765353 5712 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36203/jwks.json
I20260929 18:31:25.769981 5712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36203/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:25.770923 5712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:25.772272 5712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:25.780442 5717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.780433 5720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:25.780433 5718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:25.780634 5712 server_base.cc:1034] running on GCE node
I20260929 18:31:25.781260 5712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:25.782260 5712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:25.783440 5712 hybrid_clock.cc:648] HybridClock initialized: now 1790706685783409 us; error 42 us; skew 500 ppm
I20260929 18:31:25.785164 5712 webserver.cc:460] Webserver started at http://127.0.98.62:41533/ using document root <none> and password file <none>
I20260929 18:31:25.785728 5712 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:25.785816 5712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:25.786041 5712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:25.787962 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "b9e509b16fb0479b88c1f75a28685d08"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "2a4dbe864cb27428e7157cc6b5d7eb37"
server_key_iv: "9f0f4dba2b2e6877126bf8cd84880e6c"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.788693 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9e509b16fb0479b88c1f75a28685d08"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "2a4dbe864cb27428e7157cc6b5d7eb37"
server_key_iv: "9f0f4dba2b2e6877126bf8cd84880e6c"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.792299 5712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:31:25.794690 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:25.795841 5712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:25.795982 5712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "b9e509b16fb0479b88c1f75a28685d08"
format_stamp: "Formatted at 2026-09-29 18:31:25 on dist-test-slave-fmv8"
server_key: "2a4dbe864cb27428e7157cc6b5d7eb37"
server_key_iv: "9f0f4dba2b2e6877126bf8cd84880e6c"
server_key_version: "encryptionkey@0"
I20260929 18:31:25.796095 5712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:25.818892 5712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:25.819584 5712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:25.819788 5712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:25.826606 5712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36817
I20260929 18:31:25.826625 5778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36817 every 8 connection(s)
I20260929 18:31:25.827508 5712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:25.829738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5712
I20260929 18:31:25.829854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:25.830135 392 external_mini_cluster.cc:1428] Setting key 006794ac66985e02cd3f56ec9ffdc11d
I20260929 18:31:25.831053 5779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:25.836802 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08: Bootstrap starting.
I20260929 18:31:25.839114 5779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:25.839345 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.831460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.98.62:36817)
0929 18:31:25.832104 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.832122 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.832235 (+ 113us) server_negotiation.cc:407] Connection header received
0929 18:31:25.832947 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.832964 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.833349 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.833681 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:25.834114 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.835059 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.835960 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.836499 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.836669 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.837622 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:25.837626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:25.837638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:25.838025 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:25.838080 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:25.838105 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:25.838218 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:25.838364 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:31:25.838605 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":139,"threads_started":1}
I20260929 18:31:25.839915 5779 log.cc:824] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:25.842022 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08: No bootstrap required, opened a new log
I20260929 18:31:25.842260 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:31:25.845695 5779 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260929 18:31:25.845928 5779 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:25.846012 5779 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e509b16fb0479b88c1f75a28685d08, State: Initialized, Role: FOLLOWER
I20260929 18:31:25.846508 5779 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260929 18:31:25.846638 5779 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:25.846725 5779 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:25.846830 5779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:25.847680 5779 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260929 18:31:25.847967 5779 leader_election.cc:302] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9e509b16fb0479b88c1f75a28685d08; no voters:
I20260929 18:31:25.848269 5779 leader_election.cc:288] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:25.848395 5784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:25.848628 5784 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [term 1 LEADER]: Becoming Leader. State: Replica: b9e509b16fb0479b88c1f75a28685d08, State: Running, Role: LEADER
I20260929 18:31:25.848934 5784 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260929 18:31:25.849454 5779 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:25.850481 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e509b16fb0479b88c1f75a28685d08. Latest consensus state: current_term: 1 leader_uuid: "b9e509b16fb0479b88c1f75a28685d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260929 18:31:25.850607 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.850873 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9e509b16fb0479b88c1f75a28685d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e509b16fb0479b88c1f75a28685d08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260929 18:31:25.850972 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:25.850970 5793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:25.854120 5793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:25.859412 5793 catalog_manager.cc:1347] Generated new cluster ID: 49c50cf71d0342c0b4601e9e96715b40
I20260929 18:31:25.859807 5793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:25.867599 5793 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:25.868757 5793 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:25.875236 5793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9e509b16fb0479b88c1f75a28685d08: Generated new TSK 0
I20260929 18:31:25.875882 5793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:25.889294 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:25.883969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.0.98.62:36817)
0929 18:31:25.884095 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:25.884100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:25.884293 (+ 193us) server_negotiation.cc:407] Connection header received
0929 18:31:25.884416 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:25.884419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:25.884464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:25.884557 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:31:25.884823 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.885353 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.886587 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.886821 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.886907 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.887390 (+ 483us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:31:25.888883 (+ 1493us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:31:25.889112 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:31:25.889364 5782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:31:25.891179 5808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:25.883842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.0.1:47946)
0929 18:31:25.884176 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:25.884192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:25.884308 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:25.884585 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:25.884589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:25.884597 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:31:25.884713 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.884720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.885478 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.885481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.886460 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:25.886470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:25.886938 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:25.886961 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:25.887048 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:25.887076 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:31:25.889074 (+ 1998us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:31:25.890994 (+ 1920us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1656,"thread_start_us":103,"threads_started":1}
I20260929 18:31:25.891868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5712
E20260929 18:31:25.900228 5809 webserver.cc:533] Webserver: sslize failed:
E20260929 18:31:25.900290 5809 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:31:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (285 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:25Z Disabled control of system clock
I20260929 18:31:25.938153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46805
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46805
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/testchainca.cert with env {}
W20260929 18:31:26.050851 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:26.051146 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:26.051187 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:26.055335 5813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:26.055394 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:26.055410 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:26.055428 5813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:26.055442 5813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:31:26.055502 5813 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:31:26.055528 5813 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:31:26.061244 5813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:26.062353 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:26.063433 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:26.072345 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:26.072332 5813 server_base.cc:1034] running on GCE node
W20260929 18:31:26.072315 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:26.072551 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:26.073173 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:26.074369 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:26.075654 5813 hybrid_clock.cc:648] HybridClock initialized: now 1790706686075637 us; error 34 us; skew 500 ppm
I20260929 18:31:26.077575 5813 webserver.cc:460] Webserver started at http://127.0.98.62:41085/ using document root <none> and password file <none>
I20260929 18:31:26.078171 5813 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:26.078265 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:26.078492 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:26.080603 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "92adb3ed9f474ea19e4372d21708d4d5"
format_stamp: "Formatted at 2026-09-29 18:31:26 on dist-test-slave-fmv8"
server_key: "5a40c1b15a9b4c09841ad88f8b635ef3"
server_key_iv: "82d0122baed6a229c2bc60561868276b"
server_key_version: "encryptionkey@0"
I20260929 18:31:26.081188 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "92adb3ed9f474ea19e4372d21708d4d5"
format_stamp: "Formatted at 2026-09-29 18:31:26 on dist-test-slave-fmv8"
server_key: "5a40c1b15a9b4c09841ad88f8b635ef3"
server_key_iv: "82d0122baed6a229c2bc60561868276b"
server_key_version: "encryptionkey@0"
I20260929 18:31:26.084915 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:26.087436 5827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:26.088745 5813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:26.088891 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "92adb3ed9f474ea19e4372d21708d4d5"
format_stamp: "Formatted at 2026-09-29 18:31:26 on dist-test-slave-fmv8"
server_key: "5a40c1b15a9b4c09841ad88f8b635ef3"
server_key_iv: "82d0122baed6a229c2bc60561868276b"
server_key_version: "encryptionkey@0"
I20260929 18:31:26.089010 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:26.105898 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:26.106673 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:26.106881 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:26.113605 5813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46805
I20260929 18:31:26.113656 5879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46805 every 8 connection(s)
I20260929 18:31:26.114552 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:26.117558 5880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:26.122727 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5: Bootstrap starting.
I20260929 18:31:26.123997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5813
I20260929 18:31:26.124125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:26.124421 392 external_mini_cluster.cc:1428] Setting key 706aeb9b70b16623ae30f2a5a14974d9
I20260929 18:31:26.125381 5880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:26.126164 5880 log.cc:824] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:26.128002 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5: No bootstrap required, opened a new log
I20260929 18:31:26.131309 5880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } }
I20260929 18:31:26.131505 5880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:26.131562 5880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92adb3ed9f474ea19e4372d21708d4d5, State: Initialized, Role: FOLLOWER
I20260929 18:31:26.132014 5880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } }
I20260929 18:31:26.132172 5880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:26.132244 5880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:26.132350 5880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:26.133165 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:26.125466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.98.62:46805)
0929 18:31:26.126073 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:26.126087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:26.126118 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:31:26.126885 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:26.126908 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:26.127255 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:26.127569 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:26.127972 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.128995 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.129905 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.130360 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.130552 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:26.131384 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:26.131389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:26.131401 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:26.131858 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:26.131912 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:26.131931 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:26.132032 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:26.132219 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:31:26.132497 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260929 18:31:26.133152 5880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } }
I20260929 18:31:26.133524 5880 leader_election.cc:302] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92adb3ed9f474ea19e4372d21708d4d5; no voters:
I20260929 18:31:26.133839 5880 leader_election.cc:288] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:26.133939 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:26.134166 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [term 1 LEADER]: Becoming Leader. State: Replica: 92adb3ed9f474ea19e4372d21708d4d5, State: Running, Role: LEADER
I20260929 18:31:26.134476 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } }
I20260929 18:31:26.134943 5880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:26.136234 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:31:26.136266 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92adb3ed9f474ea19e4372d21708d4d5. Latest consensus state: current_term: 1 leader_uuid: "92adb3ed9f474ea19e4372d21708d4d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } } }
I20260929 18:31:26.136368 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:26.136626 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92adb3ed9f474ea19e4372d21708d4d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92adb3ed9f474ea19e4372d21708d4d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46805 } } }
I20260929 18:31:26.136713 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:26.136971 5894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:26.139940 5894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:26.145092 5894 catalog_manager.cc:1347] Generated new cluster ID: aaf46b70ee504c68a156384adbda155c
I20260929 18:31:26.145155 5894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:26.160943 5894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:26.161916 5894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:26.167500 5894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92adb3ed9f474ea19e4372d21708d4d5: Generated new TSK 0
I20260929 18:31:26.168200 5894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:26.213168 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:26.178956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.0.98.62:46805)
0929 18:31:26.179098 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:26.179103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:26.179312 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:31:26.179445 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:26.179448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:26.179503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:26.179617 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:31:26.179814 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.180675 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.181505 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.181853 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.181952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:26.182210 (+ 258us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:31:26.212707 (+ 30497us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:31:26.213019 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51316: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:31:26.213271 5883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51316: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:31:26.213363 5909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:26.178876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46805 (local address 127.0.0.1:51316)
0929 18:31:26.179225 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:26.179237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:26.179332 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:26.179621 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:26.179624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:26.179630 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:31:26.179718 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:26.179723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.180793 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.180796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:26.181401 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:26.181408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:26.181947 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:26.181966 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:26.182027 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:26.182047 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:31:26.212981 (+ 30934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:31:26.213153 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260929 18:31:26.214140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5813
2026-09-29T18:31:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (302 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:26 dist-test-slave-fmv8 krb5kdc[5921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:26 dist-test-slave-fmv8 krb5kdc[5921](info): set up 2 sockets
Sep 29 18:31:26 dist-test-slave-fmv8 krb5kdc[5921](info): commencing operation
krb5kdc: starting...
W20260929 18:31:27.488492 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:27 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:27.609079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:27.716952 5934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:27.717190 5934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:27.717227 5934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:27.721144 5934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:27.721192 5934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:27.721211 5934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:27.721227 5934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:27.721242 5934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:27.726182 5934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:27.727046 5934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:27.727936 5934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:27.736085 5939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:27.736100 5942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:27.736135 5934 server_base.cc:1034] running on GCE node
W20260929 18:31:27.736093 5940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:27.736793 5934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:27.737877 5934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:27.739079 5934 hybrid_clock.cc:648] HybridClock initialized: now 1790706687739051 us; error 41 us; skew 500 ppm
Sep 29 18:31:27 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:27.742221 5934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:27.743108 5934 webserver.cc:460] Webserver started at http://127.0.98.62:39549/ using document root <none> and password file <none>
I20260929 18:31:27.743654 5934 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:27.743712 5934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:27.744023 5934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:27.746024 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "be17f56545174b668bc7cc6d93f34894"
format_stamp: "Formatted at 2026-09-29 18:31:27 on dist-test-slave-fmv8"
server_key: "15d347695586a2e12a35b7463f2e7716"
server_key_iv: "8a20235a6b661c697416ff76d5d18c97"
server_key_version: "encryptionkey@0"
I20260929 18:31:27.746572 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "be17f56545174b668bc7cc6d93f34894"
format_stamp: "Formatted at 2026-09-29 18:31:27 on dist-test-slave-fmv8"
server_key: "15d347695586a2e12a35b7463f2e7716"
server_key_iv: "8a20235a6b661c697416ff76d5d18c97"
server_key_version: "encryptionkey@0"
I20260929 18:31:27.750237 5934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:27.752609 5949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:27.753844 5934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:27.753985 5934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "be17f56545174b668bc7cc6d93f34894"
format_stamp: "Formatted at 2026-09-29 18:31:27 on dist-test-slave-fmv8"
server_key: "15d347695586a2e12a35b7463f2e7716"
server_key_iv: "8a20235a6b661c697416ff76d5d18c97"
server_key_version: "encryptionkey@0"
I20260929 18:31:27.754118 5934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:27.764955 5934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:27.768000 5934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:27.768256 5934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:27.775369 5934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36497
I20260929 18:31:27.775663 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36497 every 8 connection(s)
I20260929 18:31:27.776415 5934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:27.779160 6002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:27.784588 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: Bootstrap starting.
I20260929 18:31:27.784703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 5934
I20260929 18:31:27.784827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:27.785100 392 external_mini_cluster.cc:1428] Setting key 3ff96d437fac88cb001f9d6c15045d3c
I20260929 18:31:27.787060 6002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:27.787806 6002 log.cc:824] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:27.789820 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: No bootstrap required, opened a new log
Sep 29 18:31:27 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:27.794356 6002 raft_consensus.cc:348] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260929 18:31:27.794600 6002 raft_consensus.cc:374] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:27.794657 6002 raft_consensus.cc:729] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be17f56545174b668bc7cc6d93f34894, State: Initialized, Role: FOLLOWER
I20260929 18:31:27.795130 6002 consensus_queue.cc:260] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260929 18:31:27.795275 6002 raft_consensus.cc:388] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:27.795361 6002 raft_consensus.cc:482] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:27.795459 6002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:27.796360 6002 raft_consensus.cc:504] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260929 18:31:27.796689 6002 leader_election.cc:302] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be17f56545174b668bc7cc6d93f34894; no voters:
I20260929 18:31:27.797024 6002 leader_election.cc:288] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:27.797130 6007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:27.797406 6007 raft_consensus.cc:686] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [term 1 LEADER]: Becoming Leader. State: Replica: be17f56545174b668bc7cc6d93f34894, State: Running, Role: LEADER
I20260929 18:31:27.797782 6007 consensus_queue.cc:237] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260929 18:31:27.798511 6002 sys_catalog.cc:564] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:27.800462 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:27.786377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.98.62:36497)
0929 18:31:27.786764 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:27.786779 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:27.786812 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:31:27.787463 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:27.787481 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:27.787814 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:27.788135 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:27.788988 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:27.789988 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:27.791132 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:27.791619 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:27.791747 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:27.794654 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:27.794659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:27.794694 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:27.794727 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:27.796699 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:27.797235 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:27.797243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:27.797249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:27.797385 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:27.797729 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:27.797732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:27.797734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:27.798242 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:27.798426 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:27.799075 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 18:31:27.799410 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260929 18:31:27.801412 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be17f56545174b668bc7cc6d93f34894. Latest consensus state: current_term: 1 leader_uuid: "be17f56545174b668bc7cc6d93f34894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } } }
I20260929 18:31:27.801537 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:27.801771 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be17f56545174b668bc7cc6d93f34894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be17f56545174b668bc7cc6d93f34894" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } } }
I20260929 18:31:27.801862 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894 [sys.catalog]: This master's current role is: LEADER
W20260929 18:31:27.803550 6022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:27.803609 6022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:27.803694 6023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:27.804512 6023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:27.809715 6023 catalog_manager.cc:1347] Generated new cluster ID: 1823a95ab8844ec493186257c516fdf7
I20260929 18:31:27.809803 6023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:27.814612 6023 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:27.815806 6023 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:27.827733 6023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be17f56545174b668bc7cc6d93f34894: Generated new TSK 0
I20260929 18:31:27.828498 6023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:27.864241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:27.974900 6028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:27.975179 6028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:27.975224 6028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:27.979346 6028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:27.979409 6028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:27.979501 6028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:27.984875 6028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:27.986038 6028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:27.987079 6028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:27.997156 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:27.997152 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:27.997362 6028 server_base.cc:1034] running on GCE node
W20260929 18:31:27.997152 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:27.997915 6028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:27.998631 6028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:27.999779 6028 hybrid_clock.cc:648] HybridClock initialized: now 1790706687999746 us; error 46 us; skew 500 ppm
Sep 29 18:31:27 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.003015 6028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:28.003940 6028 webserver.cc:460] Webserver started at http://127.0.98.1:40327/ using document root <none> and password file <none>
I20260929 18:31:28.004578 6028 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:28.004671 6028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:28.004909 6028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:28.006971 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "f01aa5e8b20f4d8189954e2a3531e414"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "48b9dba25a6bda11667bcc1c34dce6da"
server_key_iv: "f745cf241895935334ade4c5c8c9b157"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.007530 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f01aa5e8b20f4d8189954e2a3531e414"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "48b9dba25a6bda11667bcc1c34dce6da"
server_key_iv: "f745cf241895935334ade4c5c8c9b157"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.011221 6028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:28.013650 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.014829 6028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:28.014967 6028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "f01aa5e8b20f4d8189954e2a3531e414"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "48b9dba25a6bda11667bcc1c34dce6da"
server_key_iv: "f745cf241895935334ade4c5c8c9b157"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.015075 6028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:28.032246 6028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:28.035410 6028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:28.035679 6028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:28.036392 6028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:28.037256 6028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:28.037328 6028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.037400 6028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:28.037449 6028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.046501 6028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37127
I20260929 18:31:28.046525 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37127 every 8 connection(s)
I20260929 18:31:28.047443 6028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:28.050580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6028
I20260929 18:31:28.050717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:28.051119 392 external_mini_cluster.cc:1428] Setting key 6293f1887041f03b4c51e6361ef6ccf0
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:31:28.063464 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.049317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38361 (local address 127.0.98.62:36497)
0929 18:31:28.049468 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.049473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.050325 (+ 852us) server_negotiation.cc:407] Connection header received
0929 18:31:28.052193 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.052197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.052248 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.052358 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.053940 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.054480 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.056321 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.056554 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.056633 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.059688 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.059691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.059694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.059716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.061371 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.061880 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.061886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.061891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.061985 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.062325 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.062330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.062331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.062514 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.062617 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.063173 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.063298 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:31:28.064651 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.049745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.1:38361)
0929 18:31:28.050177 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:28.050218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:28.051947 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:28.052534 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:28.052541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:28.053026 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:28.053734 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.053756 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.054617 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.054623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.056184 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.056198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.056711 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.056718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.056971 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.057704 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:28.057722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:28.059504 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:28.061494 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.061500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.061517 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.061765 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.062077 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.062081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.062083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.062219 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.062626 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:28.062635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:28.062932 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 18:31:28.063123 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 18:31:28.063465 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260929 18:31:28.065985 6157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260929 18:31:28.066262 6157 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:28.066751 6157 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
I20260929 18:31:28.068356 5966 ts_manager.cc:194] Registered new tserver with Master: f01aa5e8b20f4d8189954e2a3531e414 (127.0.98.1:37127)
I20260929 18:31:28.070053 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:28.079205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:28.187872 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:28.188134 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:28.188248 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:28.192248 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:28.192303 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:28.192360 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:28.197516 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:28.198606 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:28.199649 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:28.208788 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:28.208904 6162 server_base.cc:1034] running on GCE node
W20260929 18:31:28.208791 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:28.208930 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:28.209379 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:28.210078 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:28.211238 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790706688211224 us; error 27 us; skew 500 ppm
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.217914 6162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:28.218897 6162 webserver.cc:460] Webserver started at http://127.0.98.2:37481/ using document root <none> and password file <none>
I20260929 18:31:28.219460 6162 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:28.219551 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:28.219781 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:28.221809 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "55be0b17448f4851a0fc67828784b4cb"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "64ddb22c56087958a7fdddc2668963fa"
server_key_iv: "ad172207c1347c0b7a6395efb5b51099"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.222362 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "55be0b17448f4851a0fc67828784b4cb"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "64ddb22c56087958a7fdddc2668963fa"
server_key_iv: "ad172207c1347c0b7a6395efb5b51099"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.225960 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:28.228345 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.229570 6162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:28.229707 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "55be0b17448f4851a0fc67828784b4cb"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "64ddb22c56087958a7fdddc2668963fa"
server_key_iv: "ad172207c1347c0b7a6395efb5b51099"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.229811 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:28.245255 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:28.248420 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:28.248631 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:28.249217 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:28.250007 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:28.250082 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.250151 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:28.250187 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.259030 6162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34895
I20260929 18:31:28.259053 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34895 every 8 connection(s)
I20260929 18:31:28.259938 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:28.265275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6162
I20260929 18:31:28.265365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:28.265645 392 external_mini_cluster.cc:1428] Setting key 4ef798067c2253728dd7f7e84ca349d0
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:28.274466 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.261739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56275 (local address 127.0.98.62:36497)
0929 18:31:28.261877 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.261882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.262751 (+ 869us) server_negotiation.cc:407] Connection header received
0929 18:31:28.264096 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.264098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.264144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.264491 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.265707 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.266254 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.267691 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.268174 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.268294 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.270873 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.270876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.270878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.270898 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.272445 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.272973 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.272977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.272979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.273057 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.273389 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.273394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.273396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.273572 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.273670 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.274171 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.274288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:28.275278 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.262181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.2:56275)
0929 18:31:28.262605 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:28.262646 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:28.263873 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:28.264657 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:28.264663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:28.265027 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:28.265561 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.265577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.266380 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.266384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.267508 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.267522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.268369 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.268375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.268569 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.269156 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:28.269173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:28.270704 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:28.272591 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.272597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.272611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.272866 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.273154 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.273157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.273159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.273297 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.273680 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:28.273688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:28.273932 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 18:31:28.274132 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 18:31:28.274451 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260929 18:31:28.276567 6291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260929 18:31:28.276772 6291 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:28.277240 6291 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:31:28.278407 5966 ts_manager.cc:194] Registered new tserver with Master: 55be0b17448f4851a0fc67828784b4cb (127.0.98.2:34895)
I20260929 18:31:28.279307 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:28.293236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:28.401213 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:28.401470 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:28.401540 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:28.405484 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:28.405555 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:28.405640 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:28.410671 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:28.411645 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:28.412657 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:28.422039 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:28.422048 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:28.422044 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:28.422361 6296 server_base.cc:1034] running on GCE node
I20260929 18:31:28.422787 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:28.423357 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:28.424548 6296 hybrid_clock.cc:648] HybridClock initialized: now 1790706688424530 us; error 37 us; skew 500 ppm
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.427167 6296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:28.428023 6296 webserver.cc:460] Webserver started at http://127.0.98.3:43317/ using document root <none> and password file <none>
I20260929 18:31:28.428647 6296 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:28.428733 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:28.428933 6296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:28.430832 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9015e6984cc4118b3642af3e5c50c2c"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "7cfdf9b90af708a2ba75343dcdd8140f"
server_key_iv: "0db15b1cc00e31a4b7bae19d722c50c7"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.431375 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9015e6984cc4118b3642af3e5c50c2c"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "7cfdf9b90af708a2ba75343dcdd8140f"
server_key_iv: "0db15b1cc00e31a4b7bae19d722c50c7"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.434963 6296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:31:28.437250 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.438457 6296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:28.438588 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "d9015e6984cc4118b3642af3e5c50c2c"
format_stamp: "Formatted at 2026-09-29 18:31:28 on dist-test-slave-fmv8"
server_key: "7cfdf9b90af708a2ba75343dcdd8140f"
server_key_iv: "0db15b1cc00e31a4b7bae19d722c50c7"
server_key_version: "encryptionkey@0"
I20260929 18:31:28.438670 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:28.487890 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:28.491353 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:28.491515 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:28.492045 6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:28.492883 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:28.492929 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.492970 6296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:28.492988 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:28.502466 6296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33937
I20260929 18:31:28.502507 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33937 every 8 connection(s)
I20260929 18:31:28.503348 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:28.509637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6296
I20260929 18:31:28.509745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:28.510082 392 external_mini_cluster.cc:1428] Setting key 56d7d39320dd2288905f1e17e7f23e25
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:28.517272 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.505147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39757 (local address 127.0.98.62:36497)
0929 18:31:28.505312 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.505317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.506115 (+ 798us) server_negotiation.cc:407] Connection header received
0929 18:31:28.507407 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.507410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.507455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.507531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.508713 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.509228 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.510790 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.511111 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.511232 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.513744 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.513747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.513749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.513772 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.515205 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.515724 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.515727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.515729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.515813 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.516222 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.516230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.516232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.516481 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.516562 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.517039 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.517150 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:31:28.518056 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.505515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.3:39757)
0929 18:31:28.505960 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:28.506006 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:28.507224 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:28.507694 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:28.507701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:28.508056 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:28.508584 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.508598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.509338 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.509340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.510679 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:28.510690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.511258 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.511263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:28.511440 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.512050 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:28.512071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:28.513582 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:28.515319 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.515324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.515335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.515597 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.515905 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:28.515907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:28.515908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:28.516045 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:28.516579 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:28.516584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:28.516804 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 18:31:28.516965 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 18:31:28.517207 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260929 18:31:28.519142 6425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260929 18:31:28.519369 6425 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:28.519837 6425 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
I20260929 18:31:28.520926 5966 ts_manager.cc:194] Registered new tserver with Master: d9015e6984cc4118b3642af3e5c50c2c (127.0.98.3:33937)
I20260929 18:31:28.521482 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39757
I20260929 18:31:28.524070 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/test-admin.keytab.
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.540701 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:31:28.543831 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:28.542213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.98.1:37127)
0929 18:31:28.542553 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.542572 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.542601 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:31:28.542696 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.542702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.542950 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.543116 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.543653 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38440: BlockingRecv error: recv got EOF from 127.0.0.1:38440 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":119,"threads_started":1}
W20260929 18:31:28.543927 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.545128 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.546169 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:28.555838 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.547092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.0.98.1:37127)
0929 18:31:28.547320 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.547323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.547336 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:28.547369 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.547371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.547403 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.547471 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.548064 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.548984 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.549822 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.550183 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.550267 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.552596 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.552604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.552617 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.552639 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.554099 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.554523 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.554529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.554530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.554607 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.554896 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.554899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.554900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.555123 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.555271 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.555491 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.555687 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
W20260929 18:31:28.559893 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:28.558920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.0.98.1:37127)
0929 18:31:28.559057 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.559060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.559102 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:31:28.559213 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.559215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.559246 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.559309 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.559758 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38454: BlockingRecv error: recv got EOF from 127.0.0.1:38454 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:31:28.559914 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.561018 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.561865 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:28.570650 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.562729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.0.98.1:37127)
0929 18:31:28.562879 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.562883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.562900 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:31:28.562989 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.562991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.563025 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.563094 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.563555 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.564059 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.564918 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.565132 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.565212 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.567703 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.567706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.567708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.567731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.569149 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.569564 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.569566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.569567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.569636 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.569863 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.569865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.569866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.570047 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.570141 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.570345 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.570477 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 18:31:28.573376 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:28.572498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.0.98.1:37127)
0929 18:31:28.572623 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.572626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.572680 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:31:28.572766 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.572768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.572799 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.572863 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.573290 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38470: BlockingRecv error: recv got EOF from 127.0.0.1:38470 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 18:31:28.573661 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706655775522-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.574678 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:28.575507 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:31:28.583787 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:28.576283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.0.98.1:37127)
0929 18:31:28.576416 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:28.576421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:28.576565 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:31:28.576632 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:28.576634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:28.576667 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:28.576733 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:28.577175 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.577693 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.578474 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:28.578682 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:28.578772 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:28.580983 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:28.580986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:28.580988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:28.581009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:28.582381 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.582733 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.582735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.582736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.582803 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:28.583057 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:28.583059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:28.583061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:28.583213 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:28.583308 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:28.583509 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:31:28.583617 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:31:28.584848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6028
I20260929 18:31:28.590436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6162
I20260929 18:31:28.595448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6296
I20260929 18:31:28.601042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 5934
2026-09-29T18:31:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2146 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[6445](info): set up 2 sockets
Sep 29 18:31:28 dist-test-slave-fmv8 krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260929 18:31:29.636583 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:29 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 18:31:29.755595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:29.864217 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:29.864485 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:29.864526 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:29.868423 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:29.868471 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:29.868490 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:29.868506 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:29.868521 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:29.873584 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:29.874511 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:29.875429 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:29.884018 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:29.884045 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:29.884044 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:29.884497 6458 server_base.cc:1034] running on GCE node
I20260929 18:31:29.884986 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:29.886045 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:29.887238 6458 hybrid_clock.cc:648] HybridClock initialized: now 1790706689887195 us; error 48 us; skew 500 ppm
Sep 29 18:31:29 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:29.889954 6458 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 18:31:29.890822 6458 webserver.cc:460] Webserver started at http://127.0.98.62:35407/ using document root <none> and password file <none>
I20260929 18:31:29.891338 6458 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:29.891395 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:29.891552 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:29.893527 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "f014287fe07a46a3ac00202f3b42bd1e"
format_stamp: "Formatted at 2026-09-29 18:31:29 on dist-test-slave-fmv8"
server_key: "54ef1fee44b7978891452e29fed675f2"
server_key_iv: "bb5675265a58014acacec7119465cac5"
server_key_version: "encryptionkey@0"
I20260929 18:31:29.894007 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "f014287fe07a46a3ac00202f3b42bd1e"
format_stamp: "Formatted at 2026-09-29 18:31:29 on dist-test-slave-fmv8"
server_key: "54ef1fee44b7978891452e29fed675f2"
server_key_iv: "bb5675265a58014acacec7119465cac5"
server_key_version: "encryptionkey@0"
I20260929 18:31:29.897565 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:29.899887 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:29.901239 6458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:29.901409 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "f014287fe07a46a3ac00202f3b42bd1e"
format_stamp: "Formatted at 2026-09-29 18:31:29 on dist-test-slave-fmv8"
server_key: "54ef1fee44b7978891452e29fed675f2"
server_key_iv: "bb5675265a58014acacec7119465cac5"
server_key_version: "encryptionkey@0"
I20260929 18:31:29.901522 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:29.918524 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:29.921548 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:29.921756 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:29.928649 6458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35157
I20260929 18:31:29.928923 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35157 every 8 connection(s)
I20260929 18:31:29.929638 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:29.931423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6458
I20260929 18:31:29.931548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:29.931849 392 external_mini_cluster.cc:1428] Setting key 7ec535c46e9dbda2bb6f0403d4fc5fd8
I20260929 18:31:29.932896 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:29.939415 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Bootstrap starting.
Sep 29 18:31:29 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:29.942060 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:29.942972 6526 log.cc:824] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:29.945377 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: No bootstrap required, opened a new log
I20260929 18:31:29.946521 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:29.933227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.0.98.62:35157)
0929 18:31:29.933615 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:29.933636 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:29.933670 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:31:29.934318 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:29.934334 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:29.934643 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:29.934912 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:29.935803 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:29.936844 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:29.937994 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:29.938457 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:29.938580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:29.941336 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:29.941342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:29.941356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:29.941386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:29.943367 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:29.943875 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:29.943881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:29.943887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:29.944021 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:29.944425 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:29.944429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:29.944430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:29.944901 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:29.945088 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:29.945327 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:31:29.945621 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":135,"threads_started":1}
I20260929 18:31:29.949465 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } }
I20260929 18:31:29.949662 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:29.949703 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f014287fe07a46a3ac00202f3b42bd1e, State: Initialized, Role: FOLLOWER
I20260929 18:31:29.950095 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } }
I20260929 18:31:29.950212 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:29.950254 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:29.950321 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:29.951143 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } }
I20260929 18:31:29.951493 6526 leader_election.cc:302] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f014287fe07a46a3ac00202f3b42bd1e; no voters:
I20260929 18:31:29.951807 6526 leader_election.cc:288] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:29.951946 6531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:29.952225 6531 raft_consensus.cc:686] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 LEADER]: Becoming Leader. State: Replica: f014287fe07a46a3ac00202f3b42bd1e, State: Running, Role: LEADER
I20260929 18:31:29.952556 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } }
I20260929 18:31:29.953006 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:29.954123 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f014287fe07a46a3ac00202f3b42bd1e. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } }
I20260929 18:31:29.954252 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:29.954520 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } }
I20260929 18:31:29.954612 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:29.954643 6540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:29.957690 6540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:29.963308 6540 catalog_manager.cc:1347] Generated new cluster ID: e8d4949f4ad04495873ad892e172b2c0
I20260929 18:31:29.963399 6540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:30.005178 6540 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:30.006327 6540 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:30.015343 6540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Generated new TSK 0
I20260929 18:31:30.016319 6540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 18:31:30.049010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:30.159428 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:30.159701 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:30.159792 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:30.163753 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:30.163842 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:30.163932 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:30.169050 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:30.170017 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:30.171000 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:30.180692 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:30.180660 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.180727 6552 server_base.cc:1034] running on GCE node
W20260929 18:31:30.180667 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.181293 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:30.181934 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:30.183130 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790706690183107 us; error 34 us; skew 500 ppm
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:30.186110 6552 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:31:30.187167 6552 webserver.cc:460] Webserver started at http://127.0.98.1:37453/ using document root <none> and password file <none>
I20260929 18:31:30.187767 6552 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:30.187865 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:30.188117 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:30.190274 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "c72cfe0c28df455e93bb2bf568e32887"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "74bee2ed7e7877a90177379d401142a8"
server_key_iv: "7ad2e750228fd3b261a3c7acbf355341"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.190886 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c72cfe0c28df455e93bb2bf568e32887"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "74bee2ed7e7877a90177379d401142a8"
server_key_iv: "7ad2e750228fd3b261a3c7acbf355341"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.194759 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:31:30.197273 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.198516 6552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:30.198665 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "c72cfe0c28df455e93bb2bf568e32887"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "74bee2ed7e7877a90177379d401142a8"
server_key_iv: "7ad2e750228fd3b261a3c7acbf355341"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.198786 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:30.220326 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:30.223397 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:30.223621 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:30.224299 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:30.225179 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:30.225265 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.225365 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:30.225421 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.234733 6552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44643
I20260929 18:31:30.234753 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44643 every 8 connection(s)
I20260929 18:31:30.235651 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:30.245577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6552
I20260929 18:31:30.245708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:30.246085 392 external_mini_cluster.cc:1428] Setting key 5e94c8c754525d832b5d1db76a3b6882
I20260929 18:31:30.249727 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.237479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52929 (local address 127.0.98.62:35157)
0929 18:31:30.237626 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.237631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.238504 (+ 873us) server_negotiation.cc:407] Connection header received
0929 18:31:30.239916 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.239919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.239965 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.240040 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.241252 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.241775 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.242950 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.243169 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.243244 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.245727 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.245733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.245737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.245765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.247548 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.248135 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.248140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.248141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.248275 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.248636 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.248641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.248643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.248831 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.248948 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.249423 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.249548 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:31:30.250523 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.237893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:52929)
0929 18:31:30.238344 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.238390 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.239664 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.240258 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.240265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.240632 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.241119 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.241133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.241888 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.241890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.242842 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.242851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.243314 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.243317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.243458 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.243975 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.243992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.245555 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.247678 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.247684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.247698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.247992 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.248378 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.248384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.248386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.248527 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.248943 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.248949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.249183 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.249380 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.249696 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260929 18:31:30.251829 6681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:30.252068 6681 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:30.252629 6681 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:30.254370 6490 ts_manager.cc:194] Registered new tserver with Master: c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:30.256006 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 18:31:30.272560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:30.378314 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:30.378533 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:30.378571 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:30.382439 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:30.382489 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:30.382542 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:30.387450 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:30.388393 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:30.389292 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:30.398332 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:30.398274 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.398407 6686 server_base.cc:1034] running on GCE node
W20260929 18:31:30.398276 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.398888 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:30.399571 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:30.400744 6686 hybrid_clock.cc:648] HybridClock initialized: now 1790706690400727 us; error 28 us; skew 500 ppm
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:30.403721 6686 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:31:30.404632 6686 webserver.cc:460] Webserver started at http://127.0.98.2:45015/ using document root <none> and password file <none>
I20260929 18:31:30.405280 6686 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:30.405387 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:30.405616 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:30.407784 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "01c6f48a2b024d48a5f3810649df84da"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "a917b4b5f78aa19dad21c4f67f1b0dea"
server_key_iv: "f37b795caa52e5d784087817e5580281"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.408386 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01c6f48a2b024d48a5f3810649df84da"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "a917b4b5f78aa19dad21c4f67f1b0dea"
server_key_iv: "f37b795caa52e5d784087817e5580281"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.412052 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:31:30.414537 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.415804 6686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:30.415941 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "01c6f48a2b024d48a5f3810649df84da"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "a917b4b5f78aa19dad21c4f67f1b0dea"
server_key_iv: "f37b795caa52e5d784087817e5580281"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.416047 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:30.449004 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:30.451854 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:30.452059 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:30.452699 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:30.453601 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:30.453687 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.453743 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:30.453776 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.463238 6686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43359
I20260929 18:31:30.463258 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43359 every 8 connection(s)
I20260929 18:31:30.464224 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:30.468406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6686
I20260929 18:31:30.468523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:30.468822 392 external_mini_cluster.cc:1428] Setting key 833d9e9fdda08bb7870beedc553127c0
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:30.478760 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.465925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46841 (local address 127.0.98.62:35157)
0929 18:31:30.466065 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.466069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.466914 (+ 845us) server_negotiation.cc:407] Connection header received
0929 18:31:30.468225 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.468231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.468297 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.468408 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.469642 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.470440 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.471556 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.471958 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.472089 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.474923 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.474927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.474930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.474956 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.476623 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.477196 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.477202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.477206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.477298 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.477674 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.477679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.477681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.477883 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.477995 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.478542 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.478643 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 18:31:30.479851 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.466333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:46841)
0929 18:31:30.466765 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.466806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.467955 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.468609 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.468620 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.469019 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.469499 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.469517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.470558 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.470560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.471429 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.471438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.472128 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.472133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.472348 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.472998 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.473015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.474750 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.476776 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.476782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.476798 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.477069 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.477409 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.477413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.477415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.477583 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.478006 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.478015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.478274 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.478486 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.478781 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260929 18:31:30.481096 6815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:30.481354 6815 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:30.481842 6815 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:30.482879 6490 ts_manager.cc:194] Registered new tserver with Master: 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359)
I20260929 18:31:30.483479 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:31:30.494345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:30.603106 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:30.603440 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:30.603528 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:30.609620 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:30.609675 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:30.609737 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:30.614753 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:30.615610 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:30.616567 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:30.625887 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:30.625849 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:30.625869 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.626154 6820 server_base.cc:1034] running on GCE node
I20260929 18:31:30.626602 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:30.627256 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:30.628408 6820 hybrid_clock.cc:648] HybridClock initialized: now 1790706690628376 us; error 46 us; skew 500 ppm
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:30.630949 6820 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:31:30.631791 6820 webserver.cc:460] Webserver started at http://127.0.98.3:36119/ using document root <none> and password file <none>
I20260929 18:31:30.632391 6820 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:30.632477 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:30.632704 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:30.634608 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "45a46a137b46439a93c69259ea1c4b05"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "26bedd7e4589762d81cf5d300d7770fb"
server_key_iv: "a5f1a93c7d085bba4cd6a1d286398383"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.635150 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45a46a137b46439a93c69259ea1c4b05"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "26bedd7e4589762d81cf5d300d7770fb"
server_key_iv: "a5f1a93c7d085bba4cd6a1d286398383"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.638654 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:30.641028 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.642122 6820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:30.642256 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "45a46a137b46439a93c69259ea1c4b05"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "26bedd7e4589762d81cf5d300d7770fb"
server_key_iv: "a5f1a93c7d085bba4cd6a1d286398383"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.642359 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:30.680958 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:30.684379 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:30.684573 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:30.685173 6820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:30.685984 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:30.686055 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.686126 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:30.686177 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.694809 6820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44111
I20260929 18:31:30.694829 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44111 every 8 connection(s)
I20260929 18:31:30.695667 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:30.700496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6820
I20260929 18:31:30.700636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:30.700948 392 external_mini_cluster.cc:1428] Setting key 0c94f7546fa35c07abe5771a275d5ad1
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:30.709070 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.697367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44199 (local address 127.0.98.62:35157)
0929 18:31:30.697488 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.697492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.698239 (+ 747us) server_negotiation.cc:407] Connection header received
0929 18:31:30.699502 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.699505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.699559 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.699634 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.700937 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.701565 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.702813 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.703071 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.703172 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.705686 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.705690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.705692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.705746 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.707109 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.707589 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.707594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.707596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.707671 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.708000 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.708005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.708006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.708229 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.708332 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.708799 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.708911 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:31:30.709798 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.697678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:44199)
0929 18:31:30.698080 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.698125 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.699291 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.699790 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.699796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.700133 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.700786 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.700802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.701696 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.701701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.702688 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.702697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.703229 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.703234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.703395 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.704016 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.704032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.705528 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.707228 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.707233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.707246 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.707489 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.707774 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.707777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.707779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.707909 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.708330 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.708337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.708567 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.708748 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.709041 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260929 18:31:30.710954 6949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:30.711146 6949 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:30.711542 6949 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:30.712519 6490 ts_manager.cc:194] Registered new tserver with Master: 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:30.713064 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44199
I20260929 18:31:30.714768 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 18:31:30.741226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33867
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35157,127.0.98.61:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:30.850340 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:30.850682 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:30.850790 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:30.855062 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:30.855144 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:30.855180 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:30.855216 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:30.855252 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:30.860554 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35157,127.0.98.61:33867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
W20260929 18:31:30.860801 6954 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35157,127.0.98.61:33867'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:31:30.861642 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:30.862651 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:30.871075 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:30.871095 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.871292 6954 server_base.cc:1034] running on GCE node
W20260929 18:31:30.871497 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:30.872103 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:30.873220 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:30.874394 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790706690874357 us; error 47 us; skew 500 ppm
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:30.877178 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 18:31:30.878080 6954 webserver.cc:460] Webserver started at http://127.0.98.61:45029/ using document root <none> and password file <none>
I20260929 18:31:30.878597 6954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:30.878659 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:30.878821 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:30.880928 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/instance:
uuid: "9f3e07eb334c435c9685ae619f44d5c7"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "564735b3e6d84d818cb27a78dfa3c5b1"
server_key_iv: "faa6769a6c6dbdaa795a58b248e614b8"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.881563 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal/instance:
uuid: "9f3e07eb334c435c9685ae619f44d5c7"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "564735b3e6d84d818cb27a78dfa3c5b1"
server_key_iv: "faa6769a6c6dbdaa795a58b248e614b8"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.885304 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 18:31:30.887594 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:30.888988 6954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:30.889119 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "9f3e07eb334c435c9685ae619f44d5c7"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "564735b3e6d84d818cb27a78dfa3c5b1"
server_key_iv: "faa6769a6c6dbdaa795a58b248e614b8"
server_key_version: "encryptionkey@0"
I20260929 18:31:30.889230 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:30.906391 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:30.909649 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:30.909855 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:30.916980 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33867
I20260929 18:31:30.917225 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33867 every 8 connection(s)
I20260929 18:31:30.918020 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:31:30.920857 7022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:30.924566 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:31:30.927219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 6954
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:31:30.939859 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.926364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.0.98.62:35157)
0929 18:31:30.926497 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.926501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.927272 (+ 771us) server_negotiation.cc:407] Connection header received
0929 18:31:30.928373 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.928377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.928438 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.928498 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.930091 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.930608 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.932056 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.932470 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.932574 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.935497 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.935501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.935506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.935533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.937324 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.937942 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.937949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.937952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.938037 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.938778 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.938781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.938783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.939016 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.939122 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.939562 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.939684 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:31:30.940972 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.928142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.98.61:33867)
0929 18:31:30.928559 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.928571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.928612 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:31:30.928763 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.928771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.928984 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.929182 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.930005 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.931501 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.932517 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.932997 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.933095 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.936140 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.936167 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.936181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.936208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.938026 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.938512 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.938516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.938518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.938614 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.938914 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.938917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.938919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.939340 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.939542 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.939771 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.940204 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":215,"spinlock_wait_cycles":5888,"thread_start_us":122,"threads_started":1}
I20260929 18:31:30.940971 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.926685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47088)
0929 18:31:30.927122 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.927162 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.928062 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.928783 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.928791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.929156 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.929905 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.929926 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.930748 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.930752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.931928 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.931938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.932599 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.932603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.932794 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.933482 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.933497 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.935356 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.937434 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.937443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.937452 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.937833 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.938526 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.938529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.938531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.938680 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.939119 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.939125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.939317 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.940187 (+ 870us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.940342 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":300,"spinlock_wait_cycles":4864,"thread_start_us":117,"threads_started":1}
I20260929 18:31:30.942224 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:31:30.950721 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.942707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46372)
0929 18:31:30.942865 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.942880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.942950 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.943374 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.943377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.943544 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.943650 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.943656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.944414 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.944418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.945552 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.945561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.945992 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.945994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.946077 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.946503 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.946527 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.947715 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.949289 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.949292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.949293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.949476 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.949881 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.949883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.949885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.949963 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.950292 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.950294 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.950365 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.950448 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.950537 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 18:31:30.950735 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.942733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.98.61:33867)
0929 18:31:30.943151 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.943160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.943171 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:31:30.943209 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.943212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.943248 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.943337 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.943759 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.944314 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.945652 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.945870 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.945947 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.947829 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.947832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.947834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.947853 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.949187 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.949557 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.949560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.949561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.949626 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.950054 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.950056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.950057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.950205 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.950288 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.950502 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.950583 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":94,"threads_started":1}
I20260929 18:31:30.953504 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Bootstrap starting.
I20260929 18:31:30.955854 7022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:30.956696 7022 log.cc:824] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:30.958514 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: No bootstrap required, opened a new log
I20260929 18:31:30.961385 7022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } }
I20260929 18:31:30.961578 7022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:30.961611 7022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3e07eb334c435c9685ae619f44d5c7, State: Initialized, Role: FOLLOWER
I20260929 18:31:30.962011 7022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } }
I20260929 18:31:30.962420 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } } }
I20260929 18:31:30.962574 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:30.963094 7022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:31:30.968627 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:30.968719 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:30.970796 6954 master_runner.cc:428] Detected that this master 9f3e07eb334c435c9685ae619f44d5c7 is joining an existing cluster
I20260929 18:31:30.970855 6954 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33867
I20260929 18:31:30.981817 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.972888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.98.61:33867)
0929 18:31:30.973037 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.973041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.973250 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:31:30.973392 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.973394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.973428 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.973473 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.974029 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.974809 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.976414 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.976679 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.976835 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.978122 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.978130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.978132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.978150 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.979988 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.980550 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.980557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.980559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.980641 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.980955 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.980959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.980961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.981168 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.981264 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.981544 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.981682 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":77}
I20260929 18:31:30.981825 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.972895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46388)
0929 18:31:30.973177 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.973191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.973301 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.973591 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.973594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.973815 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.973926 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.973933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.974918 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.974922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.976307 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.976317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.976842 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.976845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.976934 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.977553 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.977567 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.977985 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.980101 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.980105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.980107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.980412 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.980737 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.980740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.980742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.980820 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.981251 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.981254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.981352 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.981488 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.981629 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":104,"threads_started":1}
I20260929 18:31:30.982461 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.972815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.98.62:35157)
0929 18:31:30.972939 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.972945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.973257 (+ 312us) server_negotiation.cc:407] Connection header received
0929 18:31:30.973376 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.973379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.973421 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.973514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.973976 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.974798 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.977512 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.977855 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.977967 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.979095 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.979101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.979105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.979140 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.980783 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.981192 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.981196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.981198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.981283 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.981549 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.981552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.981554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.981746 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.981822 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.982193 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.982323 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:31:30.982537 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.972764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47104)
0929 18:31:30.973163 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.973175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.973293 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.973526 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.973529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.973744 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.973857 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.973863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.976440 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.976443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.977390 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.977399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.977977 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.977981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.978081 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.978544 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.978558 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.978956 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.980890 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.980894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.980898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.981098 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.981360 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.981363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.981365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.981443 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.981892 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.981895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.981983 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.982107 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.982361 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
Sep 29 18:31:30 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:31:30.998111 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.987655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.0.98.61:33867)
0929 18:31:30.987781 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:30.987785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:30.988534 (+ 749us) server_negotiation.cc:407] Connection header received
0929 18:31:30.988719 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:30.988721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:30.988762 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:30.988851 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:30.989643 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.990144 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.991403 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.991652 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.991796 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.994495 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:30.994499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:30.994502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:30.994527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:30.996206 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.996829 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.996835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.996837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.996914 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:30.997190 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:30.997193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:30.997195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:30.997393 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:30.997501 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:30.997828 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 18:31:30.997943 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:31:30.998139 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:30.987614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46396)
0929 18:31:30.988400 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:30.988452 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:30.988585 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:30.988908 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:30.988914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:30.989370 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:30.989504 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.989511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.990251 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.990255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.991268 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:30.991279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:30.991813 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:30.991819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:30.991937 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:30.992522 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:30.992536 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:30.994361 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:30.996319 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.996331 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.996341 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.996675 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.997016 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:30.997018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:30.997019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:30.997086 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:30.997501 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:30.997509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:30.997627 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:31:30.997786 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 18:31:30.997984 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":82,"threads_started":1}
I20260929 18:31:30.999863 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33867
}
attrs {
promote: true
}
}
dest_uuid: "f014287fe07a46a3ac00202f3b42bd1e"
cas_config_opid_index: -1
I20260929 18:31:31.000613 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } }
I20260929 18:31:31.002278 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } }
I20260929 18:31:31.003230 6996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 18:31:31.004714 6996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:31:31.005416 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:31.005692 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33867
I20260929 18:31:31.005999 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } }
I20260929 18:31:31.006093 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.006151 6954 master.cc:537] Master@127.0.98.61:33867 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
W20260929 18:31:31.008262 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 18:31:31.009330 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e -> Peer 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61:33867): Couldn't send request to peer 9f3e07eb334c435c9685ae619f44d5c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:31:31.010862 6954 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:31.011145 6954 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:31.011255 6954 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: stopping tablet replica
I20260929 18:31:31.026229 6954 master.cc:552] Master@127.0.98.61:33867 shutdown complete.
I20260929 18:31:31.026279 6954 master_runner.cc:305] Clearing existing system tablet
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 18:31:31.026572 6954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.026626 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.027453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34943
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:31:31.027907 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.028895 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.029107 6954 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.029240 6954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:31:31.029275 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "9f3e07eb334c435c9685ae619f44d5c7"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "564735b3e6d84d818cb27a78dfa3c5b1"
server_key_iv: "faa6769a6c6dbdaa795a58b248e614b8"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.031347 6954 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:31.033672 6954 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:31:31.033773 6954 log.cc:1196] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:31:31.034060 6954 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Deleting consensus metadata
I20260929 18:31:31.034222 6954 master_runner.cc:315] Copying system tablet from 127.0.98.62:35157
I20260929 18:31:31.035326 6954 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35157
I20260929 18:31:31.044520 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.035981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47110)
0929 18:31:31.036345 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.036359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.036475 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.036808 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.036811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.037016 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.037150 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.037157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.038338 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.038341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.039263 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.039272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.039852 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.039856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.039948 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.040533 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.040548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.040964 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.042944 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.042948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.042950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.043196 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.043477 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.043480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.043482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.043570 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.043965 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.043968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.044060 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.044194 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.044331 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":83,"threads_started":1}
I20260929 18:31:31.044605 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.036110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.0.98.62:35157)
0929 18:31:31.036293 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.036300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.036450 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:31:31.036580 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.036585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.036658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.036777 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.037286 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.038218 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.039382 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.039736 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.039837 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.041120 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.041124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.041127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.041154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.042834 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.043296 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.043299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.043301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.043385 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.043666 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.043670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.043671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.043871 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.043964 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.044261 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.044392 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:31:31.045486 6510 tablet_copy_service.cc:140] P f014287fe07a46a3ac00202f3b42bd1e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9f3e07eb334c435c9685ae619f44d5c7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47110)
I20260929 18:31:31.045665 6510 tablet_copy_service.cc:161] P f014287fe07a46a3ac00202f3b42bd1e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9f3e07eb334c435c9685ae619f44d5c7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47110: session id = 9f3e07eb334c435c9685ae619f44d5c7-00000000000000000000000000000000
I20260929 18:31:31.047436 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:31:31.049121 6954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:31:31.051405 6954 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: tablet copy: Starting download of 0 data blocks...
I20260929 18:31:31.051674 6954 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: tablet copy: Starting download of 1 WAL segments...
I20260929 18:31:31.053401 6954 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:31:31.055292 6510 tablet_copy_service.cc:342] P f014287fe07a46a3ac00202f3b42bd1e: Request end of tablet copy session 9f3e07eb334c435c9685ae619f44d5c7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47110
I20260929 18:31:31.055402 6510 tablet_copy_service.cc:433] P f014287fe07a46a3ac00202f3b42bd1e: ending tablet copy session 9f3e07eb334c435c9685ae619f44d5c7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9f3e07eb334c435c9685ae619f44d5c7
I20260929 18:31:31.062808 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:31.064405 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.064467 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.065143 6954 server_base.cc:1034] running on GCE node
W20260929 18:31:31.065277 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.065471 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.065766 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.066908 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790706691066896 us; error 29 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.068693 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 18:31:31.069144 6954 webserver.cc:460] Webserver started at http://127.0.98.61:39469/ using document root <none> and password file <none>
I20260929 18:31:31.069329 6954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.069371 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.070639 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.071681 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.071942 6954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.072013 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "9f3e07eb334c435c9685ae619f44d5c7"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "564735b3e6d84d818cb27a78dfa3c5b1"
server_key_iv: "faa6769a6c6dbdaa795a58b248e614b8"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.072090 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.078703 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.080891 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.085458 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33867
I20260929 18:31:31.085999 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33867 every 8 connection(s)
I20260929 18:31:31.086184 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:31:31.087558 7129 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:31:31.088465 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Bootstrap starting.
I20260929 18:31:31.096052 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:31:31.096396 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: Bootstrap complete.
I20260929 18:31:31.096866 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } }
I20260929 18:31:31.096998 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f3e07eb334c435c9685ae619f44d5c7, State: Initialized, Role: LEARNER
I20260929 18:31:31.097133 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } }
I20260929 18:31:31.097467 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } }
I20260929 18:31:31.097617 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:31:31.097640 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:31.100957 7143 catalog_manager.cc:1259] Loaded cluster ID: e8d4949f4ad04495873ad892e172b2c0
I20260929 18:31:31.101009 7143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: loading cluster ID for follower catalog manager: success
I20260929 18:31:31.102720 7143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: acquiring CA information for follower catalog manager: success
I20260929 18:31:31.103875 7143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:31:31.172657 7058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:31.172895 7058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:31.172967 7058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:31.176988 7058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:31.177045 7058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:31.177065 7058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:31.177083 7058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:31.177098 7058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:31.182174 7058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:31.183177 7058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:31.184221 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:31.192339 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.192382 7058 server_base.cc:1034] running on GCE node
W20260929 18:31:31.192327 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.192332 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.193053 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.194212 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.195406 7058 hybrid_clock.cc:648] HybridClock initialized: now 1790706691195391 us; error 27 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.197992 7058 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 18:31:31.199060 7058 webserver.cc:460] Webserver started at http://127.0.98.60:44729/ using document root <none> and password file <none>
I20260929 18:31:31.199678 7058 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.199769 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.199975 7058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:31.202019 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/instance:
uuid: "c1693b3cb4464c13877cc60f68e9d78f"
format_stamp: "Formatted at 2026-09-29 18:31:31 on dist-test-slave-fmv8"
server_key: "4a61778ea558cecfe845f31aae24697d"
server_key_iv: "aed881d4615b059137822ff683feb54a"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.202638 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal/instance:
uuid: "c1693b3cb4464c13877cc60f68e9d78f"
format_stamp: "Formatted at 2026-09-29 18:31:31 on dist-test-slave-fmv8"
server_key: "4a61778ea558cecfe845f31aae24697d"
server_key_iv: "aed881d4615b059137822ff683feb54a"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.206600 7058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:31:31.209076 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.210415 7058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:31:31.210565 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "c1693b3cb4464c13877cc60f68e9d78f"
format_stamp: "Formatted at 2026-09-29 18:31:31 on dist-test-slave-fmv8"
server_key: "4a61778ea558cecfe845f31aae24697d"
server_key_iv: "aed881d4615b059137822ff683feb54a"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.210727 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.233485 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.236769 7058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:31.236956 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.244956 7058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34943
I20260929 18:31:31.245222 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34943 every 8 connection(s)
I20260929 18:31:31.245930 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:31:31.248708 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:31.252838 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:31:31.254590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7058
I20260929 18:31:31.258461 6681 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.267406 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.255356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.0.98.62:35157)
0929 18:31:31.255476 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.255480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.256018 (+ 538us) server_negotiation.cc:407] Connection header received
0929 18:31:31.257479 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.257481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.257528 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.257658 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.258549 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.259231 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.260454 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.260670 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.260783 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.263374 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.263377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.263379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.263399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.265218 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.265751 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.265755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.265760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.265844 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.266232 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.266235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.266237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.266485 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.266577 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.267136 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.267253 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:31:31.268273 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.255389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47126)
0929 18:31:31.255877 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.255914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.257245 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.257698 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.257703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.258124 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.258401 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.258413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.259358 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.259362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.260337 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.260346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.260816 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.260820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.260978 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.261550 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.261575 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.263234 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.265339 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.265346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.265352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.265645 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.265938 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.265942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.265944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.266118 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.266589 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.266595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.266870 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.267069 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.267381 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
I20260929 18:31:31.268276 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.255821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.98.60:34943)
0929 18:31:31.256100 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.256110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.256143 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:31:31.257245 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.257268 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.257498 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.257678 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.258403 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.259578 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.260495 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.260843 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.260949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.263857 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.263868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.263882 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.263906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.265850 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.266349 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.266356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.266357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.266431 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.266718 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.266720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.266721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.266939 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.267136 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.267330 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.267460 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260929 18:31:31.269461 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 18:31:31.277201 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.269899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46412)
0929 18:31:31.270005 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.270020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.270089 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.270482 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.270484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.270615 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.270702 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.270707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.271539 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.271542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.272063 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.272070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.272623 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.272625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.272697 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.273044 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.273055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.274162 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.275807 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.275809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.275810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.275976 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.276270 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.276273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.276275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.276358 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.276726 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.276730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.276812 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.276928 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.277051 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 18:31:31.277233 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.270026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.0.98.61:33867)
0929 18:31:31.270289 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.270292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.270303 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:31:31.270338 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.270340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.270378 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.270449 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.270800 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.271434 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.272206 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.272520 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.272599 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.274288 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.274291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.274293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.274312 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.275710 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.276055 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.276057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.276058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.276124 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.276456 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.276459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.276461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.276625 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.276725 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.276978 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.277086 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260929 18:31:31.277900 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.284924 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.278408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.60:34943)
0929 18:31:31.278513 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.278517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.278531 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:31.278585 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.278589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.278627 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.278701 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.278981 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.279457 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.280131 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.280412 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.280484 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.282103 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.282106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.282108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.282125 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.283451 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.283846 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.283852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.283854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.283922 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.284192 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.284195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.284197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.284353 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.284451 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.284680 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.284806 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":45}
I20260929 18:31:31.284922 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.278333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35456)
0929 18:31:31.278436 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.278451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.278527 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.278722 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.278724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.278836 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.278895 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.278899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.279543 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.279545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.280041 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.280048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.280506 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.280509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.280562 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.280895 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.280905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.281967 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.283569 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.283573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.283575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.283755 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.284017 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.284019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.284021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.284095 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.284451 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.284453 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.284527 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.284631 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.284755 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:31:31.287899 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Bootstrap starting.
I20260929 18:31:31.290091 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:31.290755 7211 log.cc:824] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:31.292577 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: No bootstrap required, opened a new log
I20260929 18:31:31.295291 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 18:31:31.295470 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:31.295501 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1693b3cb4464c13877cc60f68e9d78f, State: Initialized, Role: FOLLOWER
I20260929 18:31:31.295902 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 18:31:31.296339 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 18:31:31.296471 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:31.297003 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:31:31.301741 7228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:31.301826 7228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:31.304888 7058 master_runner.cc:428] Detected that this master c1693b3cb4464c13877cc60f68e9d78f is joining an existing cluster
I20260929 18:31:31.304948 7058 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:34943
I20260929 18:31:31.311239 7217 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:31.311831 7217 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 18:31:31.312849 7217 leader_election.cc:288] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157), 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61:33867)
I20260929 18:31:31.313201 6500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1693b3cb4464c13877cc60f68e9d78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f014287fe07a46a3ac00202f3b42bd1e" is_pre_election: true
I20260929 18:31:31.313748 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1693b3cb4464c13877cc60f68e9d78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3e07eb334c435c9685ae619f44d5c7" is_pre_election: true
I20260929 18:31:31.313918 7103 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Handling vote request from an unknown peer c1693b3cb4464c13877cc60f68e9d78f
I20260929 18:31:31.314072 7103 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate c1693b3cb4464c13877cc60f68e9d78f for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 18:31:31.314766 7162 leader_election.cc:302] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1693b3cb4464c13877cc60f68e9d78f; no voters: 9f3e07eb334c435c9685ae619f44d5c7, f014287fe07a46a3ac00202f3b42bd1e
I20260929 18:31:31.315136 7217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:31.316397 7217 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:31:31.321321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6552
I20260929 18:31:31.327955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44643
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35157
--webserver_port=37453
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:31:31.334668 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.0.98.61:33867)
0929 18:31:31.307878 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.307882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.307893 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:31:31.307930 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.307935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.307981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.308065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.308639 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309438 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.310681 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.311044 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.311135 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.312650 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.312654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.312657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.312685 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.315422 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.332897 (+ 17475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.332905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.332910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.333034 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.333425 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.333430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.333433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.333716 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.333824 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.334250 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.334412 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:31:31.337141 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.98.62:35157)
0929 18:31:31.307130 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.307137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.307405 (+ 268us) server_negotiation.cc:407] Connection header received
0929 18:31:31.307592 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.307598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.307665 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.307721 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.308411 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309216 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.310400 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.310762 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.310845 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.314146 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.314151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.314156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.314184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.316915 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335416 (+ 18501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.335424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.335426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.335540 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335923 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.335928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.335932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.336229 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.336332 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.336768 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.336919 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:31.337155 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35470)
0929 18:31:31.308295 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.308309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.308410 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.308662 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.308665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.308893 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.308988 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.308994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309896 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.309898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.310744 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.310752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.311374 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.311377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.311473 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.332729 (+ 21256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.332748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.333214 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.335517 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.335522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.335749 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.335994 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.335998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.336084 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.336515 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.336517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.336586 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.336673 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.336961 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":162,"thread_start_us":66,"threads_started":1}
I20260929 18:31:31.337953 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.60:34943)
0929 18:31:31.307581 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.307585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.308361 (+ 776us) server_negotiation.cc:407] Connection header received
0929 18:31:31.308492 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.308494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.308526 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.308566 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.309089 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309788 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.310855 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.311161 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.311238 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.333375 (+ 22137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.333379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.333381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.333403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.335407 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335845 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.335848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.335850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.335914 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.336215 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.336218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.336220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.336406 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.336486 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.336797 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.337813 (+ 1016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":911,"server-negotiator.queue_time_us":36}
I20260929 18:31:31.337983 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47140)
0929 18:31:31.307321 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.307336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.307472 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.307877 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.307879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.308110 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.308273 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.308280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309350 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.309353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.310251 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.310258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.310999 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.311002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.313103 (+ 2101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.313503 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.313515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.313990 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.317060 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.317063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.317064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.335258 (+ 18194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.335679 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.335683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.335688 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.335798 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.336397 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.336401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.336524 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.337352 (+ 828us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.337525 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1979,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260929 18:31:31.338550 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.307056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46418)
0929 18:31:31.307616 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.307634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.307706 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.308109 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.308112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.308371 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.308493 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.308500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.309578 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.309581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.310522 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.310532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.311295 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.311299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.311396 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.312003 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.312019 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.312508 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.315557 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.315560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.315562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.332705 (+ 17143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.333185 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.333190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.333194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.333301 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.333891 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.333896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.334026 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.334850 (+ 824us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.338397 (+ 3547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":99,"threads_started":1}
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.350889 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.342040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35484)
0929 18:31:31.342211 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.342225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.342303 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.342579 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.342582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.342768 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.342869 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.342876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.343858 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.343861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.344757 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.344767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.345384 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.345388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.345485 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.345933 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.345950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.347518 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.349392 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.349396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.349398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.349640 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.349913 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.349916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.349919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.350006 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.350390 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.350393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.350485 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.350600 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.350729 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 18:31:31.350950 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.342104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.0.98.60:34943)
0929 18:31:31.342200 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.342204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.342267 (+ 63us) server_negotiation.cc:407] Connection header received
0929 18:31:31.342420 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.342423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.342470 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.342547 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.342976 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.343748 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.344879 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.345273 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.345370 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.347651 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.347655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.347658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.347683 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.349283 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.349741 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.349745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.349747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.349821 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.350099 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.350102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.350104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.350292 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.350384 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.350667 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.350809 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:31:31.352195 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:34943: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 34943
}
attrs {
promote: true
}
}
dest_uuid: "f014287fe07a46a3ac00202f3b42bd1e"
cas_config_opid_index: 5
I20260929 18:31:31.352854 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } }
I20260929 18:31:31.353936 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.355479 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:34943
I20260929 18:31:31.355626 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.355737 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.356101 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:31:31.356788 7058 master.cc:537] Master@127.0.98.60:34943 shutting down...
I20260929 18:31:31.356851 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:31:31.358420 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e -> Peer c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60:34943): Couldn't send request to peer c1693b3cb4464c13877cc60f68e9d78f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:31:31.361516 7058 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:31.361770 7058 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:31.361860 7058 tablet_replica.cc:330] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: stopping tablet replica
I20260929 18:31:31.363358 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.356493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.98.61:33867)
0929 18:31:31.356614 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.356618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.358750 (+ 2132us) server_negotiation.cc:407] Connection header received
0929 18:31:31.358943 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.358949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.359014 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.359078 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:31.359439 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.360211 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.361959 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.362575 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.362654 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.362803 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.363166 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.363243 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:31.363966 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.354100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46420)
0929 18:31:31.358667 (+ 4567us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.358682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.358819 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.359222 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.359225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.359242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:31.359321 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.359326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.360347 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.360349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.361730 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.361741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.362905 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.362911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.363006 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.363049 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.363777 (+ 728us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.363852 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4490}
I20260929 18:31:31.364598 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:31:31.365279 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:31:31.367158 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.369102 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f014287fe07a46a3ac00202f3b42bd1e. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.369211 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:31:31.369311 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: true } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.369410 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:31:31.369526 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: attempting to promote NON_VOTER 9f3e07eb334c435c9685ae619f44d5c7 to VOTER
I20260929 18:31:31.369908 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } }
I20260929 18:31:31.371289 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.371012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35492)
0929 18:31:31.371148 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.371214 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:31:31.371414 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 LEARNER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:31:31.372294 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.372042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35504)
0929 18:31:31.372181 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.372234 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:31:31.372377 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e -> Peer c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60:34943): Couldn't send request to peer c1693b3cb4464c13877cc60f68e9d78f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:31:31.372563 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 18:31:31.374094 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.374545 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.375582 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.375684 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.375824 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.375900 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.376300 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.376456 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:31.380954 7058 master.cc:552] Master@127.0.98.60:34943 shutdown complete.
I20260929 18:31:31.381006 7058 master_runner.cc:305] Clearing existing system tablet
I20260929 18:31:31.381237 7058 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.381291 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.382649 7058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:31:31.383580 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.383869 7058 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.384037 7058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.384087 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "c1693b3cb4464c13877cc60f68e9d78f"
format_stamp: "Formatted at 2026-09-29 18:31:31 on dist-test-slave-fmv8"
server_key: "4a61778ea558cecfe845f31aae24697d"
server_key_iv: "aed881d4615b059137822ff683feb54a"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.386190 7058 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:31.388459 7058 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:31:31.388540 7058 log.cc:1196] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:31:31.388775 7058 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Deleting consensus metadata
I20260929 18:31:31.389137 7058 master_runner.cc:315] Copying system tablet from 127.0.98.62:35157
I20260929 18:31:31.390200 7058 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35157
I20260929 18:31:31.399303 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.390827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.0.98.62:35157)
0929 18:31:31.390945 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.390949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.391094 (+ 145us) server_negotiation.cc:407] Connection header received
0929 18:31:31.391211 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.391214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.391264 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.391359 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.391867 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.392686 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.393938 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.394252 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.394360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.395580 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.395584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.395587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.395612 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.397421 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.397995 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.397999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.398001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.398084 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.398381 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.398384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.398386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.398576 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.398697 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.399027 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.399153 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:31:31.399956 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.390724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47150)
0929 18:31:31.390998 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.391012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.391130 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.391388 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.391391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.391613 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.391746 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.391753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.392820 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.392824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.393810 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.393820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.394367 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.394371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.394466 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.395005 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.395020 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.395433 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.397567 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.397570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.397573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.397832 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.398204 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.398207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.398209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.398290 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.398764 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.398767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.398855 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.399649 (+ 794us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.399789 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260929 18:31:31.400607 6510 tablet_copy_service.cc:140] P f014287fe07a46a3ac00202f3b42bd1e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c1693b3cb4464c13877cc60f68e9d78f ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47150)
I20260929 18:31:31.400701 6510 tablet_copy_service.cc:161] P f014287fe07a46a3ac00202f3b42bd1e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c1693b3cb4464c13877cc60f68e9d78f at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47150: session id = c1693b3cb4464c13877cc60f68e9d78f-00000000000000000000000000000000
I20260929 18:31:31.401706 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:31:31.402889 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:31:31.405393 7058 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: tablet copy: Starting download of 0 data blocks...
I20260929 18:31:31.405673 7058 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: tablet copy: Starting download of 1 WAL segments...
I20260929 18:31:31.407074 7058 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:31:31.408699 6510 tablet_copy_service.cc:342] P f014287fe07a46a3ac00202f3b42bd1e: Request end of tablet copy session c1693b3cb4464c13877cc60f68e9d78f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47150
I20260929 18:31:31.408771 6510 tablet_copy_service.cc:433] P f014287fe07a46a3ac00202f3b42bd1e: ending tablet copy session c1693b3cb4464c13877cc60f68e9d78f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c1693b3cb4464c13877cc60f68e9d78f
I20260929 18:31:31.415318 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:31.417025 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.417354 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.417492 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.420050 7058 server_base.cc:1034] running on GCE node
I20260929 18:31:31.420356 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.420648 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.421797 7058 hybrid_clock.cc:648] HybridClock initialized: now 1790706691421787 us; error 33 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.423621 7058 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 18:31:31.424053 7058 webserver.cc:460] Webserver started at http://127.0.98.60:33301/ using document root <none> and password file <none>
I20260929 18:31:31.424299 7058 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.424347 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.425652 7058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.426669 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.426927 7058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:31.426997 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "c1693b3cb4464c13877cc60f68e9d78f"
format_stamp: "Formatted at 2026-09-29 18:31:31 on dist-test-slave-fmv8"
server_key: "4a61778ea558cecfe845f31aae24697d"
server_key_iv: "aed881d4615b059137822ff683feb54a"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.427068 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.435654 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.437740 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.442679 7058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34943
I20260929 18:31:31.442803 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34943 every 8 connection(s)
I20260929 18:31:31.443261 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:31:31.445053 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:31:31.446164 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Bootstrap starting.
I20260929 18:31:31.457360 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:31:31.457666 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Bootstrap complete.
I20260929 18:31:31.458282 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } }
I20260929 18:31:31.458437 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c1693b3cb4464c13877cc60f68e9d78f, State: Initialized, Role: LEARNER
I20260929 18:31:31.458542 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } }
I20260929 18:31:31.458998 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:31.459321 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.459499 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:31:31.463047 7325 catalog_manager.cc:1259] Loaded cluster ID: e8d4949f4ad04495873ad892e172b2c0
I20260929 18:31:31.463150 7325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: loading cluster ID for follower catalog manager: success
I20260929 18:31:31.465222 7325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: acquiring CA information for follower catalog manager: success
I20260929 18:31:31.466675 7325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:31:31.468523 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:31.468740 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:31.468823 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:31.473011 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:31.473090 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:31.473229 7237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:31.478425 7237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37453
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:31.479501 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:31.480511 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:31:31.485585 6815 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
W20260929 18:31:31.490056 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.490163 7237 server_base.cc:1034] running on GCE node
W20260929 18:31:31.490197 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.490061 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.490677 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.491355 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.492553 7237 hybrid_clock.cc:648] HybridClock initialized: now 1790706691492520 us; error 43 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.495184 7237 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:31:31.495934 7237 webserver.cc:460] Webserver started at http://127.0.98.1:37453/ using document root <none> and password file <none>
I20260929 18:31:31.496488 7237 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.496558 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.500370 7237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:31:31.502240 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.503414 7237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:31.503561 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "c72cfe0c28df455e93bb2bf568e32887"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "74bee2ed7e7877a90177379d401142a8"
server_key_iv: "7ad2e750228fd3b261a3c7acbf355341"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.504067 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.528887 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.531888 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:31.532085 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.532697 7237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:31.533522 7237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:31.533568 7237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.533636 7237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:31.533666 7237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.542393 7237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44643
I20260929 18:31:31.542465 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44643 every 8 connection(s)
I20260929 18:31:31.543361 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:31.544255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7237
I20260929 18:31:31.544453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6686
I20260929 18:31:31.552352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43359
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35157
--webserver_port=45015
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.562234 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.546230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38421 (local address 127.0.98.62:35157)
0929 18:31:31.546389 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.546394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.547755 (+ 1361us) server_negotiation.cc:407] Connection header received
0929 18:31:31.548904 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.548909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.548964 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.549048 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.550770 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551625 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.553685 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.554011 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554104 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.557801 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.557805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.557808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.557838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.559338 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.559901 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.559908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.559911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.559995 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.560718 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.560723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.560727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.561199 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.561299 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.561919 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.562046 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:31.562825 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.549543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35065 (local address 127.0.98.61:33867)
0929 18:31:31.549678 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.549684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.549708 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:31.549746 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.549749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.549799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.549870 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.550772 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551571 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.553598 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.553935 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554011 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.557547 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.557553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.557559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.557593 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.559485 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.560220 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.560225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.560227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.560329 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.560739 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.560746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.560750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.561048 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.561153 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.562487 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.562618 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:31:31.564061 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.548942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.98.1:35065)
0929 18:31:31.549193 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.549204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.549267 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.549985 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.549989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.550160 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.550599 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.550619 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551739 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.551742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.553462 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.553473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.554178 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.554426 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.555193 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.555218 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.557336 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.559771 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.559778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.559782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.560059 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.560444 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.560449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.560454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.560612 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.561285 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.561294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.561636 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.561861 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.562246 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":364,"spinlock_wait_cycles":19584,"thread_start_us":98,"threads_started":1}
I20260929 18:31:31.564059 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.548563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.98.1:53997)
0929 18:31:31.548818 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.548852 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.548950 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.549818 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.549824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.550067 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.550599 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.550618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551941 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.551944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.553462 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.553473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.554143 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.554491 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.555237 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.555253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.559568 (+ 4315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.561762 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.561765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.561767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.562002 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.562286 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.562289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.562291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.562372 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.562827 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.562831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.562912 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.563013 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.563124 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":323,"spinlock_wait_cycles":6144,"thread_start_us":100,"threads_started":1}
I20260929 18:31:31.564059 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.546625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:38421)
0929 18:31:31.547579 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.547632 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.548677 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.549248 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.549259 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.549781 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.550612 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.550625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551760 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.551765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.553536 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.553550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.554274 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.554460 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.555384 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.555403 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.557624 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.559455 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.559461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.559477 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.559790 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.560413 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.560419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.560422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.560601 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.561361 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.561367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.561632 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.562701 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.562848 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":352,"spinlock_wait_cycles":10752,"thread_start_us":534,"threads_started":1}
I20260929 18:31:31.564941 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.549216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53997 (local address 127.0.98.60:34943)
0929 18:31:31.549494 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.549498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.549515 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:31:31.549555 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.549558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.549609 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.549694 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.550764 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.551824 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.553642 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.554003 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.554096 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.559729 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.559734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.559740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.559770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.561632 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.562106 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.562109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.562111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.562186 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.562488 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.562491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.562493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.562685 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.562785 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.564208 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.564753 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260929 18:31:31.565435 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33867
I20260929 18:31:31.565529 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34943
I20260929 18:31:31.565536 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:31.565672 7454 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.565672 7455 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.565672 7456 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.566190 7456 heartbeater.cc:511] Master 127.0.98.61:33867 requested a full tablet report, sending...
I20260929 18:31:31.566205 7454 heartbeater.cc:511] Master 127.0.98.60:34943 requested a full tablet report, sending...
I20260929 18:31:31.566205 7455 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:31.567441 6490 ts_manager.cc:194] Re-registered known tserver with Master: c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
I20260929 18:31:31.567770 7275 ts_manager.cc:194] Registered new tserver with Master: c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
I20260929 18:31:31.568163 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38421
I20260929 18:31:31.568799 7085 ts_manager.cc:194] Registered new tserver with Master: c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
W20260929 18:31:31.668112 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:31.668365 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:31.668406 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:31.672547 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:31.672608 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:31.672694 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:31.677866 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45015
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:31.678848 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:31.679845 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:31.689254 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.689447 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.689457 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.689613 7462 server_base.cc:1034] running on GCE node
I20260929 18:31:31.690001 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.690562 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.691741 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790706691691702 us; error 50 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.694643 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:31:31.695479 7462 webserver.cc:460] Webserver started at http://127.0.98.2:45015/ using document root <none> and password file <none>
I20260929 18:31:31.696012 7462 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.696125 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.699934 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:31:31.701927 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.703099 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:31.703233 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "01c6f48a2b024d48a5f3810649df84da"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "a917b4b5f78aa19dad21c4f67f1b0dea"
server_key_iv: "f37b795caa52e5d784087817e5580281"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.703732 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.715324 6949 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
I20260929 18:31:31.723323 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.726279 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:31.726478 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.727105 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:31.727991 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:31.728068 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.728140 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:31.728232 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.737903 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43359
I20260929 18:31:31.738250 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43359 every 8 connection(s)
I20260929 18:31:31.738909 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:31.739933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7462
I20260929 18:31:31.740239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6820
I20260929 18:31:31.748353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44111
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35157
--webserver_port=36119
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.756608 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.742648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59935 (local address 127.0.98.62:35157)
0929 18:31:31.742751 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.742755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.742961 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:31:31.743140 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.743146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.743209 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.743298 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.745040 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.745865 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.747837 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748230 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748334 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.752342 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.752346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.752349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.752384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.754250 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.754763 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.754767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.754769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.754855 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.755312 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.755315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.755317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.755553 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.755625 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.756314 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.756449 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:31:31.756753 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.742273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35789 (local address 127.0.98.61:33867)
0929 18:31:31.742405 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.742410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.742545 (+ 135us) server_negotiation.cc:407] Connection header received
0929 18:31:31.742974 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.742980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.743036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.743096 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.745040 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.745865 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748124 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748490 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.752796 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.752803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.752808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.752844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.754963 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.755420 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.755424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.755426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.755503 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.755827 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.755830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.755832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.756040 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.756112 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.756490 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.756609 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:31:31.758262 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.742173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.98.2:35789)
0929 18:31:31.742453 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.742466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.742735 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.743422 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.743431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.744021 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.744864 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.744885 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.746044 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.746048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.748008 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.748016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748780 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.749000 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.749716 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.749746 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.752619 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.755072 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.755074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.755076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.755311 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.755631 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.755636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.755641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.755726 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.756216 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.756220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.756314 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.756936 (+ 622us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.757169 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":276,"spinlock_wait_cycles":11520,"thread_start_us":85,"threads_started":1}
I20260929 18:31:31.758484 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.742531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:59935)
0929 18:31:31.742890 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.742904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.743018 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.743422 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.743431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.744021 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.744864 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.744885 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.746065 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.746067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.747691 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.747703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748506 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.748824 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.750228 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.750242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.752108 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.754382 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.754389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.754404 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.754658 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.754952 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.754955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.754957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.755160 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.755700 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.755706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.756007 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.756765 (+ 758us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.757218 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":541,"spinlock_wait_cycles":10368,"thread_start_us":128,"threads_started":1}
I20260929 18:31:31.759348 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.744326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58101 (local address 127.0.98.60:34943)
0929 18:31:31.744473 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.744479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.744496 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:31:31.744541 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.744546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.744614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.744697 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.745383 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.746292 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.747958 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748334 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748425 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.752736 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.752741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.752746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.752788 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.755914 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.756520 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.756526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.756530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.756626 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.757710 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.757715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.757719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.757970 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.758097 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.758961 (+ 864us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.759124 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:31:31.759548 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33867
I20260929 18:31:31.759778 7593 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.760241 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:31.760313 7593 heartbeater.cc:511] Master 127.0.98.61:33867 requested a full tablet report, sending...
I20260929 18:31:31.760339 7592 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.760489 7592 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:31.760593 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.741191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.98.2:58101)
0929 18:31:31.741644 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.741695 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.742730 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.744870 (+ 2140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.744876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.745081 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.745231 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.745241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.746438 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.746444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.747809 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.747816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.748528 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.748531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.748835 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.749764 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.749775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.752578 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.756083 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.756087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.756092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.756391 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.757475 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.757481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.757486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.757586 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.758183 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.758188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.758303 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.759790 (+ 1487us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.760007 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":202,"spinlock_wait_cycles":384,"thread_start_us":125,"threads_started":1}
I20260929 18:31:31.761365 6490 ts_manager.cc:194] Re-registered known tserver with Master: 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359)
I20260929 18:31:31.761550 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34943
I20260929 18:31:31.761582 7085 ts_manager.cc:194] Registered new tserver with Master: 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359)
I20260929 18:31:31.761628 7591 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.761768 7591 heartbeater.cc:511] Master 127.0.98.60:34943 requested a full tablet report, sending...
I20260929 18:31:31.762091 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59935
I20260929 18:31:31.762445 7275 ts_manager.cc:194] Registered new tserver with Master: 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359)
I20260929 18:31:31.798413 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.794708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35518)
0929 18:31:31.794903 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.794918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.795003 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.795287 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.795289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.795303 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:31.795384 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.795388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.796397 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.796400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.797152 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.797160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.798083 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.798086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.798160 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.798179 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.798245 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.798305 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 18:31:31.798444 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.794830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.0.98.60:34943)
0929 18:31:31.794960 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.794964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.794979 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:31.795112 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.795115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.795174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.795272 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:31.795480 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.796297 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.797280 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.797916 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.798035 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.798194 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.798259 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.798325 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:31.799154 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 18:31:31.799517 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f014287fe07a46a3ac00202f3b42bd1e. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.799690 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: LEARNER
W20260929 18:31:31.863816 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:31.864080 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:31.864123 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:31.868060 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:31.868124 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:31.868204 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:31.873610 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36119
--tserver_master_addrs=127.0.98.62:35157,127.0.98.61:33867,127.0.98.60:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:31.874555 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:31.875491 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:31:31.885154 7598 server_base.cc:1034] running on GCE node
W20260929 18:31:31.885118 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.885118 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:31.885118 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:31.886206 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:31.886814 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:31.888018 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790706691888005 us; error 24 us; skew 500 ppm
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:31.890969 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:31:31.891870 7598 webserver.cc:460] Webserver started at http://127.0.98.3:36119/ using document root <none> and password file <none>
I20260929 18:31:31.892481 7598 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:31.892584 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:31.896294 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260929 18:31:31.898373 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.899766 7598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:31.899894 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "45a46a137b46439a93c69259ea1c4b05"
format_stamp: "Formatted at 2026-09-29 18:31:30 on dist-test-slave-fmv8"
server_key: "26bedd7e4589762d81cf5d300d7770fb"
server_key_iv: "a5f1a93c7d085bba4cd6a1d286398383"
server_key_version: "encryptionkey@0"
I20260929 18:31:31.900470 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:31.912496 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:31.915630 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:31.915839 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:31.916467 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:31.917284 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:31.917356 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.917425 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:31.917469 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:31.926013 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44111
I20260929 18:31:31.926038 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44111 every 8 connection(s)
I20260929 18:31:31.927026 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:31.935588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7598
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 18:31:31 dist-test-slave-fmv8 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 18:31:31.946259 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.929251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.98.3:44189)
0929 18:31:31.929844 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.929907 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.931110 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.931805 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.931814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.932441 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.933207 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.933221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934352 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.934355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.937630 (+ 3275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.937641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.938357 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.938361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.938467 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.938913 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.938931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.940453 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.942719 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.942725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.942742 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.943141 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.943470 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.943473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.943474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.943655 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.944196 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.944205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.944514 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.944753 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.945097 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":397,"spinlock_wait_cycles":53888,"thread_start_us":634,"threads_started":1}
I20260929 18:31:31.946861 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.928804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44189 (local address 127.0.98.60:34943)
0929 18:31:31.928925 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.928929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.930011 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 18:31:31.931341 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.931345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.931398 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.931450 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.933369 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934182 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.937780 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.938096 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.938186 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.940566 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.940570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.940573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.940598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.942564 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.943268 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.943274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.943277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.943362 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.943777 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.943781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.943784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.944018 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.944115 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.946543 (+ 2428us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.946693 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:31.947189 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.930432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41695 (local address 127.0.98.61:33867)
0929 18:31:31.930570 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.930574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.930773 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:31:31.931341 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.931345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.931396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.931450 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.933368 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934190 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.936010 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.936390 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.936485 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.940810 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.940815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.940820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.940851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.942719 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.944227 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.944231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.944233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.944319 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.944975 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.944979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.944980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.945204 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.945294 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.946680 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.947014 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:31:31.946563 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.930308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.98.3:41695)
0929 18:31:31.930689 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.930701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.931110 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.931805 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.931813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.932447 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.933201 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.933221 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934352 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.934355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.935862 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.935871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.936653 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.936657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.936787 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.937587 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.937609 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.940665 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.942829 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.942832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.942834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.943464 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.944422 (+ 958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.944425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.944427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.944508 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.945336 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.945338 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.945434 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.945529 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.945652 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":340,"spinlock_wait_cycles":14848,"thread_start_us":75,"threads_started":1}
I20260929 18:31:31.949007 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.929250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:37843)
0929 18:31:31.930556 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.930569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.931112 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.931805 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.931813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.932438 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:31.933202 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.933221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934351 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.934354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.935863 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.935874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.936585 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.936588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.936791 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.938350 (+ 1559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:31.938365 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:31.940075 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:31.947524 (+ 7449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.947527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.947529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.947738 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.948033 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:31.948036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:31.948038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:31.948124 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:31.948550 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:31.948553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:31.948650 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.948726 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.948849 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":306,"spinlock_wait_cycles":27776,"thread_start_us":135,"threads_started":1}
I20260929 18:31:31.949435 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.929084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37843 (local address 127.0.98.62:35157)
0929 18:31:31.929185 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.929190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.930646 (+ 1456us) server_negotiation.cc:407] Connection header received
0929 18:31:31.931332 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.931336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.931388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.931438 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.933358 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.934183 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.935992 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.936344 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.936411 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.940621 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.940627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.940630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.940655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.944866 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.947848 (+ 2982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.947852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.947854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.947936 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.948244 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.948247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.948249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.948432 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.948513 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.949249 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.949337 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:31:31.949137 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34943
I20260929 18:31:31.949836 7727 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.950186 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35157
I20260929 18:31:31.950296 7728 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.950438 7727 heartbeater.cc:511] Master 127.0.98.60:34943 requested a full tablet report, sending...
I20260929 18:31:31.950462 7728 heartbeater.cc:511] Master 127.0.98.62:35157 requested a full tablet report, sending...
I20260929 18:31:31.949224 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33867
I20260929 18:31:31.951009 7729 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:31.951143 7729 heartbeater.cc:511] Master 127.0.98.61:33867 requested a full tablet report, sending...
I20260929 18:31:31.951505 6490 ts_manager.cc:194] Re-registered known tserver with Master: 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:31.951826 7275 ts_manager.cc:194] Registered new tserver with Master: 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:31.951889 7085 ts_manager.cc:194] Registered new tserver with Master: 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:31.952235 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37843
I20260929 18:31:31.954633 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.937849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.0.98.61:33867)
0929 18:31:31.941883 (+ 4034us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.941887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.943066 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 18:31:31.943107 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.943110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.943154 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.943235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.944086 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.944889 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.945977 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.946338 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.946432 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.947961 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.947964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.947966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.947988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.950046 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.953250 (+ 3204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.953253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.953255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.953334 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.953610 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.953613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.953615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.953849 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.953963 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.954363 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.954480 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3963,"thread_start_us":76,"threads_started":1}
I20260929 18:31:31.956328 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.937455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.0.98.62:35157)
0929 18:31:31.941910 (+ 4455us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.941916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.942270 (+ 354us) server_negotiation.cc:407] Connection header received
0929 18:31:31.942310 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.942312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.942358 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.942441 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.943415 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.944250 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.945357 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.945702 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.945787 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.951910 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.951914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.951918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.951942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.953972 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.954494 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.954499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.954500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.954591 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.955091 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.955097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.955100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.955299 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.955418 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.956030 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.956204 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3831,"thread_start_us":73,"threads_started":1}
I20260929 18:31:31.960278 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.942154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.60:34943)
0929 18:31:31.942664 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.942673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.950480 (+ 7807us) server_negotiation.cc:407] Connection header received
0929 18:31:31.950688 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.950692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.950737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.950794 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:31.953192 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.954032 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.955126 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.955474 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.955564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.957060 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:31.957063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:31.957066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:31.957088 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:31.958444 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.959004 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.959007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.959009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.959095 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:31.959363 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:31.959366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:31.959368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:31.959563 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:31.959707 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.959910 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.960100 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":84,"threads_started":1}
I20260929 18:31:31.959040 6490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:31.962687 6490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:31.965140 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: true } } }
I20260929 18:31:31.965276 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.968564 7745 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: attempting to promote NON_VOTER c1693b3cb4464c13877cc60f68e9d78f to VOTER
I20260929 18:31:31.970777 7745 consensus_queue.cc:237] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } }
I20260929 18:31:31.972508 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:31:31.972563 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 LEARNER]: Refusing update from remote peer f014287fe07a46a3ac00202f3b42bd1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:31:31.972918 7747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 18:31:31.973088 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 18:31:31.974763 7747 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.975485 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.975289 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.976082 7747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.976063 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.976192 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.976320 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.976400 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.976181 7747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:31.977746 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.977865 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:31.978091 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f014287fe07a46a3ac00202f3b42bd1e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:31.978224 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:31.985678 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.980715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44111 (local address 127.0.0.1:59704)
0929 18:31:31.981620 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.981633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.981713 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.982384 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.982386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.982401 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:31.982473 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.982477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.983548 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.983554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.984494 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.984501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.985278 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.985282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.985367 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.985393 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.985455 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.985526 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":30,"thread_start_us":91,"threads_started":1}
I20260929 18:31:31.985723 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.980855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.98.3:44111)
0929 18:31:31.981953 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.981965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.981987 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:31:31.982056 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.982061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.982252 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.982401 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:31.982590 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.983453 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.984606 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.985143 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.985280 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.985452 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.985503 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.985589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":98,"threads_started":1}
I20260929 18:31:31.986100 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.979921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43359 (local address 127.0.0.1:33782)
0929 18:31:31.980053 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.980070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.980569 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.981368 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.981371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.981388 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:31.981492 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.981501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.983233 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.983236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.984499 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.984509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.985778 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.985783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.985862 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.985882 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.985926 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.985984 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":22}
I20260929 18:31:31.986826 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.980242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.0.98.2:43359)
0929 18:31:31.980857 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.980869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.980893 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:31.980975 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.980980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.981209 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.981394 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:31.981628 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.983100 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.984821 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.985639 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.986393 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.986526 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.986576 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.986685 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":101,"threads_started":1}
I20260929 18:31:31.987288 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.980514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44643 (local address 127.0.0.1:43960)
0929 18:31:31.981224 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:31.981238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:31.981317 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:31.983103 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:31.983106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:31.983124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:31.983200 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.983205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.984544 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.984550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.985697 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:31.985708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.986745 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.986748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:31.986838 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.986859 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:31:31.987100 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 18:31:31.987171 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":119,"threads_started":1}
I20260929 18:31:31.987360 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:31.980986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.98.1:44643)
0929 18:31:31.982671 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:31.982692 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:31.982714 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:31:31.982790 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:31.982795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:31.982965 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:31.983106 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:31.983290 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.984416 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.985821 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:31.986599 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:31.986710 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:31.986830 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:31.987126 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 18:31:31.987220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":87,"threads_started":1}
I20260929 18:31:31.989275 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 961b73090f9e40c08cf8646ee63e1032 (DEFAULT_TABLE table=test-table [id=f2c6460dabc64a19bdccacd3b163ef89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:31.989280 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 961b73090f9e40c08cf8646ee63e1032 (DEFAULT_TABLE table=test-table [id=f2c6460dabc64a19bdccacd3b163ef89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:31.990309 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961b73090f9e40c08cf8646ee63e1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:31.990330 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961b73090f9e40c08cf8646ee63e1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:31.990053 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 961b73090f9e40c08cf8646ee63e1032 (DEFAULT_TABLE table=test-table [id=f2c6460dabc64a19bdccacd3b163ef89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:31.990922 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 961b73090f9e40c08cf8646ee63e1032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:31.997151 7761 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Bootstrap starting.
I20260929 18:31:31.997881 7762 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Bootstrap starting.
I20260929 18:31:31.999013 7763 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Bootstrap starting.
I20260929 18:31:31.999780 7761 tablet_bootstrap.cc:654] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:32.000535 7762 tablet_bootstrap.cc:654] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:32.000867 7763 tablet_bootstrap.cc:654] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:32.001161 7761 log.cc:824] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:32.001226 7762 log.cc:824] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:32.002154 7763 log.cc:824] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:32.003145 7762 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: No bootstrap required, opened a new log
I20260929 18:31:32.003232 7761 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: No bootstrap required, opened a new log
I20260929 18:31:32.003360 7762 ts_tablet_manager.cc:1397] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:31:32.003465 7761 ts_tablet_manager.cc:1397] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260929 18:31:32.003947 7763 tablet_bootstrap.cc:492] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: No bootstrap required, opened a new log
I20260929 18:31:32.004109 7763 ts_tablet_manager.cc:1397] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:31:32.007658 7762 raft_consensus.cc:348] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.007890 7762 raft_consensus.cc:374] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:32.007997 7762 raft_consensus.cc:729] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a46a137b46439a93c69259ea1c4b05, State: Initialized, Role: FOLLOWER
I20260929 18:31:32.008445 7762 consensus_queue.cc:260] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.008402 7761 raft_consensus.cc:348] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.008808 7761 raft_consensus.cc:374] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:32.008890 7761 raft_consensus.cc:729] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6f48a2b024d48a5f3810649df84da, State: Initialized, Role: FOLLOWER
I20260929 18:31:32.009047 7763 raft_consensus.cc:348] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.009276 7763 raft_consensus.cc:374] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:32.009341 7763 raft_consensus.cc:729] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72cfe0c28df455e93bb2bf568e32887, State: Initialized, Role: FOLLOWER
I20260929 18:31:32.009439 7762 ts_tablet_manager.cc:1428] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:31:32.009449 7761 consensus_queue.cc:260] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.009678 7728 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
I20260929 18:31:32.009765 7763 consensus_queue.cc:260] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.010427 7761 ts_tablet_manager.cc:1428] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260929 18:31:32.010792 7592 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
I20260929 18:31:32.013417 7763 ts_tablet_manager.cc:1428] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260929 18:31:32.014549 7455 heartbeater.cc:503] Master 127.0.98.62:35157 was elected leader, sending a full tablet report...
W20260929 18:31:32.046517 7457 tablet.cc:2369] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:32.094756 7768 raft_consensus.cc:482] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:32.094966 7768 raft_consensus.cc:504] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.096230 7768 leader_election.cc:288] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111), c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
I20260929 18:31:32.100361 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.096478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44111 (local address 127.0.98.2:55543)
0929 18:31:32.096888 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:32.096898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:32.096954 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:32.097253 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:32.097256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:32.097281 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:32.097386 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:32.097391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.098187 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.098190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:32.099207 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:32.099216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.099892 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:32.099992 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.100017 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:31:32.100085 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:31:32.100195 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":335}
I20260929 18:31:32.100397 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.096647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55543 (local address 127.0.98.3:44111)
0929 18:31:32.096765 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.096770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.096962 (+ 192us) server_negotiation.cc:407] Connection header received
0929 18:31:32.097015 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.097017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.097067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.097141 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:32.097514 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.098063 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099324 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.099769 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.099936 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.100121 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.100268 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:31:32.100521 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.096769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55151 (local address 127.0.98.1:44643)
0929 18:31:32.096883 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.096888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.096910 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:31:32.096985 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.096989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.097044 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.097134 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:32.097421 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.098059 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099399 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.099797 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099916 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.099979 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.100322 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.100411 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:31:32.100790 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.096478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44643 (local address 127.0.98.2:55151)
0929 18:31:32.096639 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:32.096655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:32.096755 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:32.097159 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:32.097161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:32.097176 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:32.097302 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:32.097307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.098175 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.098179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:32.099209 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:32.099218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.099935 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.099939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:32.100056 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.100103 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:31:32.100289 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 18:31:32.100356 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":64}
I20260929 18:31:32.101157 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b73090f9e40c08cf8646ee63e1032" candidate_uuid: "01c6f48a2b024d48a5f3810649df84da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a46a137b46439a93c69259ea1c4b05" is_pre_election: true
I20260929 18:31:32.101363 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b73090f9e40c08cf8646ee63e1032" candidate_uuid: "01c6f48a2b024d48a5f3810649df84da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72cfe0c28df455e93bb2bf568e32887" is_pre_election: true
I20260929 18:31:32.101436 7681 raft_consensus.cc:2454] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6f48a2b024d48a5f3810649df84da in term 0.
I20260929 18:31:32.101569 7408 raft_consensus.cc:2454] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6f48a2b024d48a5f3810649df84da in term 0.
I20260929 18:31:32.101915 7481 leader_election.cc:302] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6f48a2b024d48a5f3810649df84da, 45a46a137b46439a93c69259ea1c4b05; no voters:
I20260929 18:31:32.102185 7768 raft_consensus.cc:2781] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:32.102268 7768 raft_consensus.cc:482] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:32.102317 7768 raft_consensus.cc:3037] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:32.103778 7768 raft_consensus.cc:504] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.104233 7768 leader_election.cc:288] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [CANDIDATE]: Term 1 election: Requested vote from peers 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111), c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643)
I20260929 18:31:32.104549 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b73090f9e40c08cf8646ee63e1032" candidate_uuid: "01c6f48a2b024d48a5f3810649df84da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a46a137b46439a93c69259ea1c4b05"
I20260929 18:31:32.104559 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b73090f9e40c08cf8646ee63e1032" candidate_uuid: "01c6f48a2b024d48a5f3810649df84da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72cfe0c28df455e93bb2bf568e32887"
I20260929 18:31:32.104681 7408 raft_consensus.cc:3037] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:32.104681 7681 raft_consensus.cc:3037] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:32.105974 7408 raft_consensus.cc:2454] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6f48a2b024d48a5f3810649df84da in term 1.
I20260929 18:31:32.106000 7681 raft_consensus.cc:2454] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6f48a2b024d48a5f3810649df84da in term 1.
I20260929 18:31:32.106297 7478 leader_election.cc:302] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6f48a2b024d48a5f3810649df84da, c72cfe0c28df455e93bb2bf568e32887; no voters:
I20260929 18:31:32.106482 7768 raft_consensus.cc:2781] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:32.106722 7768 raft_consensus.cc:686] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 1 LEADER]: Becoming Leader. State: Replica: 01c6f48a2b024d48a5f3810649df84da, State: Running, Role: LEADER
I20260929 18:31:32.107136 7768 consensus_queue.cc:237] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } }
I20260929 18:31:32.110519 6489 catalog_manager.cc:5495] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c6f48a2b024d48a5f3810649df84da (127.0.98.2). New cstate: current_term: 1 leader_uuid: "01c6f48a2b024d48a5f3810649df84da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:32.142712 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.139469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.0.98.2:43359)
0929 18:31:32.139614 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.139617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.139632 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:32.139703 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.139705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.139756 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.139852 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:32.140066 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.140578 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.141413 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.141650 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.141731 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.141991 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:32.142389 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:32.142478 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.142534 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.142579 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:31:32.148237 7681 raft_consensus.cc:1261] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Refusing update from remote peer 01c6f48a2b024d48a5f3810649df84da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:32.148254 7408 raft_consensus.cc:1261] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Refusing update from remote peer 01c6f48a2b024d48a5f3810649df84da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:32.148962 7768 consensus_queue.cc:1035] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:32.149431 7770 consensus_queue.cc:1035] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:32.156333 7775 mvcc.cc:205] Tried to move back new op lower bound from 7334734611032154112 to 7334734610874122240. Current Snapshot: MvccSnapshot[applied={T|T < 7334734611032154112}]
I20260929 18:31:32.162736 7774 mvcc.cc:205] Tried to move back new op lower bound from 7334734611032154112 to 7334734610874122240. Current Snapshot: MvccSnapshot[applied={T|T < 7334734611032154112 or (T in {7334734611032154112})}]
I20260929 18:31:32.168895 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:32.169061 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:32.173257 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f014287fe07a46a3ac00202f3b42bd1e: Sending DeleteTablet for 3 replicas of tablet 961b73090f9e40c08cf8646ee63e1032
I20260929 18:31:32.173878 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b73090f9e40c08cf8646ee63e1032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33782
I20260929 18:31:32.174369 7782 tablet_replica.cc:330] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: stopping tablet replica
I20260929 18:31:32.174687 7782 raft_consensus.cc:2229] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:32.174669 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b73090f9e40c08cf8646ee63e1032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43960
I20260929 18:31:32.174736 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b73090f9e40c08cf8646ee63e1032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59704
I20260929 18:31:32.175105 7782 raft_consensus.cc:2258] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:32.175323 7783 tablet_replica.cc:330] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: stopping tablet replica
I20260929 18:31:32.175773 7784 tablet_replica.cc:330] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: stopping tablet replica
I20260929 18:31:32.176286 7784 raft_consensus.cc:2229] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:32.176517 7782 ts_tablet_manager.cc:1905] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:32.176560 7784 raft_consensus.cc:2258] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:32.177834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6458
I20260929 18:31:32.178303 7783 raft_consensus.cc:2229] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:32.178690 7783 raft_consensus.cc:2258] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:32.180583 7782 ts_tablet_manager.cc:1918] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:32.180688 7782 log.cc:1196] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/961b73090f9e40c08cf8646ee63e1032
I20260929 18:31:32.181043 7782 ts_tablet_manager.cc:1939] T 961b73090f9e40c08cf8646ee63e1032 P 01c6f48a2b024d48a5f3810649df84da: Deleting consensus metadata
I20260929 18:31:32.181370 7783 ts_tablet_manager.cc:1905] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:32.183822 7783 ts_tablet_manager.cc:1918] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:32.183900 7783 log.cc:1196] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/961b73090f9e40c08cf8646ee63e1032
I20260929 18:31:32.184211 7783 ts_tablet_manager.cc:1939] T 961b73090f9e40c08cf8646ee63e1032 P c72cfe0c28df455e93bb2bf568e32887: Deleting consensus metadata
I20260929 18:31:32.184616 7784 ts_tablet_manager.cc:1905] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:31:32.185994 7344 connection.cc:535] server connection from 127.0.0.1:43960 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:31:32.186390 7481 connection.cc:535] server connection from 127.0.0.1:33782 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:31:32.187765 7784 ts_tablet_manager.cc:1918] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:32.187892 7784 log.cc:1196] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/961b73090f9e40c08cf8646ee63e1032
I20260929 18:31:32.188323 7784 ts_tablet_manager.cc:1939] T 961b73090f9e40c08cf8646ee63e1032 P 45a46a137b46439a93c69259ea1c4b05: Deleting consensus metadata
W20260929 18:31:32.189481 7616 connection.cc:441] server connection from 127.0.0.1:59704 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59704 (request call id 1) could send its response
I20260929 18:31:32.195358 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.176989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.98.61:33867)
0929 18:31:32.177303 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.177311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.185473 (+ 8162us) server_negotiation.cc:407] Connection header received
0929 18:31:32.185676 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.185679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.185735 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.185785 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:32.186755 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.187551 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.188770 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.189122 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.189192 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.191320 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:32.191325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:32.191327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:32.191359 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:32.193442 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.193936 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.193940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.193942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.194038 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.194335 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.194339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.194340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.194565 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:32.194671 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.195057 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.195178 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 18:31:32.202343 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.191427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.98.60:34943)
0929 18:31:32.191536 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.191540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.191950 (+ 410us) server_negotiation.cc:407] Connection header received
0929 18:31:32.192117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.192120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.192197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.192247 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:32.193058 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.193831 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.194972 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.195305 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.195373 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.197303 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:32.197307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:32.197310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:32.197335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:32.199257 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.201026 (+ 1769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.201030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.201032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.201112 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.201392 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.201395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.201397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.201602 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:32.201718 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.202060 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.202184 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":768}
I20260929 18:31:32.202977 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.191104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.61:33867)
0929 18:31:32.191223 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.191228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.192077 (+ 849us) server_negotiation.cc:407] Connection header received
0929 18:31:32.192326 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.192331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.192394 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.192452 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:32.193497 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.194333 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.195794 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.196221 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.196305 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.199042 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:32.199046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:32.199050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:32.199075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:32.200732 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.201360 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.201365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.201369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.201466 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.201812 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.201815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.201817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.202062 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:32.202168 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.202758 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.202869 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:31:32.205948 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:32.177785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.98.60:34943)
0929 18:31:32.177908 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:32.177914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:32.196392 (+ 18478us) server_negotiation.cc:407] Connection header received
0929 18:31:32.196577 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:32.196580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:32.196627 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:32.196685 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:32.197505 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.198522 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.199908 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:32.200225 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:32.200348 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:32.201990 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:32.201994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:32.201997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:32.202039 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:32.204051 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.204645 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.204649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.204651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.204735 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:32.205167 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:32.205170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:32.205172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:32.205358 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:32.205476 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:32.205693 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:31:32.205800 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":9344}
I20260929 18:31:33.160422 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.159861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:40581)
0929 18:31:33.160205 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.160306 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
W20260929 18:31:33.160869 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.164248 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.163847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:55687)
0929 18:31:33.164051 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.164136 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260929 18:31:33.164474 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.164032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:43825)
0929 18:31:33.164287 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.164384 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20260929 18:31:33.164507 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
W20260929 18:31:33.164824 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.673449 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f014287fe07a46a3ac00202f3b42bd1e)
I20260929 18:31:33.673605 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } }
I20260929 18:31:33.674100 7796 leader_election.cc:288] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157), 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61:33867)
I20260929 18:31:33.674146 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f014287fe07a46a3ac00202f3b42bd1e)
I20260929 18:31:33.674353 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } }
I20260929 18:31:33.674739 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.674252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47602)
0929 18:31:33.674540 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.674626 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":66,"threads_started":1}
W20260929 18:31:33.674937 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.675532 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157), c1693b3cb4464c13877cc60f68e9d78f (127.0.98.60:34943)
I20260929 18:31:33.676025 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.675590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47626)
0929 18:31:33.675851 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.675907 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":65,"threads_started":1}
I20260929 18:31:33.676100 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.675570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47616)
0929 18:31:33.675992 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.676040 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20260929 18:31:33.676200 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
W20260929 18:31:33.676301 7262 leader_election.cc:334] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.677544 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.677321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47632)
0929 18:31:33.677434 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.677485 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 18:31:33.677693 7080 leader_election.cc:334] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.679033 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.674309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33867 (local address 127.0.0.1:46452)
0929 18:31:33.674736 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.674752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.674860 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.675160 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.675163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.675185 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.675289 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.675295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.676671 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.676674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.677995 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.678006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.678644 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.678647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.678736 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.678761 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.678833 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.678916 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":43,"thread_start_us":98,"threads_started":1}
I20260929 18:31:33.679068 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.674395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.98.61:33867)
0929 18:31:33.674668 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.674672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.674918 (+ 246us) server_negotiation.cc:407] Connection header received
0929 18:31:33.674979 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.674982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.675038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.675129 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.675416 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.676575 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.678121 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.678532 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.678614 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.678659 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.678863 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.678953 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260929 18:31:33.679442 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1693b3cb4464c13877cc60f68e9d78f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f3e07eb334c435c9685ae619f44d5c7" is_pre_election: true
I20260929 18:31:33.679590 7103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1693b3cb4464c13877cc60f68e9d78f in term 1.
I20260929 18:31:33.679872 7261 leader_election.cc:302] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f3e07eb334c435c9685ae619f44d5c7, c1693b3cb4464c13877cc60f68e9d78f; no voters: f014287fe07a46a3ac00202f3b42bd1e
I20260929 18:31:33.680138 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.675688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:35548)
0929 18:31:33.676299 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.676312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.676393 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.676817 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.676819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.676833 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.676896 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.676908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.677874 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.677877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.678846 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.678855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.679740 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.679744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.679840 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.679857 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.679947 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.680017 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":54,"threads_started":1}
I20260929 18:31:33.680138 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.676002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.0.98.60:34943)
0929 18:31:33.676584 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.676592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.676605 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:33.676670 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.676673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.676713 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.676798 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.677000 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.677739 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.678956 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.679590 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.679690 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.679765 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.679964 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.680030 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":524,"thread_start_us":95,"threads_started":1}
I20260929 18:31:33.680119 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:31:33.680235 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f014287fe07a46a3ac00202f3b42bd1e)
I20260929 18:31:33.680295 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:31:33.681322 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } }
I20260929 18:31:33.681638 7796 leader_election.cc:288] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 election: Requested vote from peers f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157), 9f3e07eb334c435c9685ae619f44d5c7 (127.0.98.61:33867)
I20260929 18:31:33.681918 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f3e07eb334c435c9685ae619f44d5c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c1693b3cb4464c13877cc60f68e9d78f" is_pre_election: true
I20260929 18:31:33.681995 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1693b3cb4464c13877cc60f68e9d78f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f3e07eb334c435c9685ae619f44d5c7"
I20260929 18:31:33.682054 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.681830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47644)
0929 18:31:33.681942 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.681994 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:31:33.682106 7103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:31:33.682175 7285 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f3e07eb334c435c9685ae619f44d5c7 in current term 2: Already voted for candidate c1693b3cb4464c13877cc60f68e9d78f in this term.
I20260929 18:31:33.682578 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f3e07eb334c435c9685ae619f44d5c7; no voters: c1693b3cb4464c13877cc60f68e9d78f, f014287fe07a46a3ac00202f3b42bd1e
I20260929 18:31:33.682818 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.682614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47660)
0929 18:31:33.682716 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.682763 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:31:33.682941 7262 leader_election.cc:334] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
I20260929 18:31:33.683251 7103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1693b3cb4464c13877cc60f68e9d78f in term 2.
I20260929 18:31:33.683360 7797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:31:33.683553 7261 leader_election.cc:302] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f3e07eb334c435c9685ae619f44d5c7, c1693b3cb4464c13877cc60f68e9d78f; no voters: f014287fe07a46a3ac00202f3b42bd1e
I20260929 18:31:33.683740 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:31:33.683943 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [term 2 LEADER]: Becoming Leader. State: Replica: c1693b3cb4464c13877cc60f68e9d78f, State: Running, Role: LEADER
I20260929 18:31:33.684226 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } }
I20260929 18:31:33.685487 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1693b3cb4464c13877cc60f68e9d78f. Latest consensus state: current_term: 2 leader_uuid: "c1693b3cb4464c13877cc60f68e9d78f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:33.685601 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:33.685874 7808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:33.687975 7808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:33.688457 7808 catalog_manager.cc:1259] Loaded cluster ID: e8d4949f4ad04495873ad892e172b2c0
I20260929 18:31:33.688536 7808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:33.689031 7808 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:33.689363 7808 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Loaded TSK: 0
I20260929 18:31:33.690007 7808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:33.711186 7274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:33.714203 7274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:33.717103 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.716834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47676)
0929 18:31:33.716979 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.717032 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:31:33.717121 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [term 2 FOLLOWER]: Refusing update from remote peer c1693b3cb4464c13877cc60f68e9d78f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 18:31:33.717592 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 18:31:33.717779 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.717549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.0.1:47684)
0929 18:31:33.717656 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.717701 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:31:33.717878 7262 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f -> Peer f014287fe07a46a3ac00202f3b42bd1e (127.0.98.62:35157): Couldn't send request to peer f014287fe07a46a3ac00202f3b42bd1e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:31:33.719055 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1693b3cb4464c13877cc60f68e9d78f. Latest consensus state: current_term: 2 leader_uuid: "c1693b3cb4464c13877cc60f68e9d78f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:33.719187 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:33.720206 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c1693b3cb4464c13877cc60f68e9d78f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:33.720323 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:33.720623 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c1693b3cb4464c13877cc60f68e9d78f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f014287fe07a46a3ac00202f3b42bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35157 } } peers { permanent_uuid: "9f3e07eb334c435c9685ae619f44d5c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33867 } attrs { promote: false } } peers { permanent_uuid: "c1693b3cb4464c13877cc60f68e9d78f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attrs { promote: false } } }
I20260929 18:31:33.720784 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3e07eb334c435c9685ae619f44d5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:31:33.730470 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.726397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43359 (local address 127.0.0.1:33802)
0929 18:31:33.726472 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.726484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.726557 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.727406 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.727408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.727427 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.727500 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.727505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.728463 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.728467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.729215 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.729229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.730075 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.730079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.730175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.730202 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.730252 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.730328 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":2}
I20260929 18:31:33.730708 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.726137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44643 (local address 127.0.0.1:43962)
0929 18:31:33.726242 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.726255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.726335 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.727056 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.727059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.727074 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.727177 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.727184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.728286 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.728290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.729455 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.729463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.730324 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.730330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.730441 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.730467 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.730535 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.730600 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:31:33.730787 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.726270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.0.98.1:44643)
0929 18:31:33.726773 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.726780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.726796 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:31:33.726844 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.726849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.726894 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.727046 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.727305 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.728185 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.729576 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.730188 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.730324 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.730399 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.730572 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.730657 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":96,"threads_started":1}
I20260929 18:31:33.731065 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.726645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.0.98.2:43359)
0929 18:31:33.726897 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.726903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.726919 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:31:33.726985 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.726988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.727039 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.727184 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.727630 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.728360 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.729347 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.729932 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.730746 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.730807 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.730845 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.730935 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260929 18:31:33.731576 7525 tablet_service.cc:1467] Processing CreateTablet for tablet c16f02f4c202451a893a274294a13abc (DEFAULT_TABLE table=test-table [id=67a39ccdb4ee440fbaa2dd87e93a0994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:33.731693 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.726834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44111 (local address 127.0.0.1:59706)
0929 18:31:33.727709 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.727720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.727825 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.728074 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.728076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.728091 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.728196 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.728202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.729240 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.729243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.730298 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.730308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.731293 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.731297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.731389 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.731426 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.731506 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.731571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":103,"threads_started":1}
I20260929 18:31:33.731709 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.727258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.0.98.3:44111)
0929 18:31:33.727473 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.727478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.727793 (+ 315us) server_negotiation.cc:407] Connection header received
0929 18:31:33.727904 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.727907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.727967 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.728053 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.728338 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.729117 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.730438 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.731092 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.731185 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.731241 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.731521 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.731591 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260929 18:31:33.731760 7388 tablet_service.cc:1467] Processing CreateTablet for tablet c16f02f4c202451a893a274294a13abc (DEFAULT_TABLE table=test-table [id=67a39ccdb4ee440fbaa2dd87e93a0994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:33.731853 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16f02f4c202451a893a274294a13abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:33.731988 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16f02f4c202451a893a274294a13abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:33.734328 7819 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Bootstrap starting.
I20260929 18:31:33.734498 7661 tablet_service.cc:1467] Processing CreateTablet for tablet c16f02f4c202451a893a274294a13abc (DEFAULT_TABLE table=test-table [id=67a39ccdb4ee440fbaa2dd87e93a0994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:33.734741 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16f02f4c202451a893a274294a13abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:33.735183 7819 tablet_bootstrap.cc:654] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:33.735710 7820 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: Bootstrap starting.
I20260929 18:31:33.736380 7819 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: No bootstrap required, opened a new log
I20260929 18:31:33.736577 7820 tablet_bootstrap.cc:654] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:33.736730 7819 ts_tablet_manager.cc:1397] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:33.737185 7822 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Bootstrap starting.
I20260929 18:31:33.737296 7819 raft_consensus.cc:348] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.737430 7819 raft_consensus.cc:374] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:33.737468 7819 raft_consensus.cc:729] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6f48a2b024d48a5f3810649df84da, State: Initialized, Role: FOLLOWER
I20260929 18:31:33.737591 7819 consensus_queue.cc:260] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.737977 7820 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: No bootstrap required, opened a new log
I20260929 18:31:33.737987 7819 ts_tablet_manager.cc:1428] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:33.738060 7820 ts_tablet_manager.cc:1397] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:33.738121 7822 tablet_bootstrap.cc:654] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:33.739176 7820 raft_consensus.cc:348] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.739274 7820 raft_consensus.cc:374] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:33.739305 7820 raft_consensus.cc:729] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72cfe0c28df455e93bb2bf568e32887, State: Initialized, Role: FOLLOWER
I20260929 18:31:33.739444 7820 consensus_queue.cc:260] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.739884 7820 ts_tablet_manager.cc:1428] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:33.740434 7591 heartbeater.cc:503] Master 127.0.98.60:34943 was elected leader, sending a full tablet report...
I20260929 18:31:33.741201 7822 tablet_bootstrap.cc:492] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: No bootstrap required, opened a new log
I20260929 18:31:33.741283 7822 ts_tablet_manager.cc:1397] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:31:33.741685 7822 raft_consensus.cc:348] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.741781 7822 raft_consensus.cc:374] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:33.741808 7822 raft_consensus.cc:729] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a46a137b46439a93c69259ea1c4b05, State: Initialized, Role: FOLLOWER
I20260929 18:31:33.741904 7822 consensus_queue.cc:260] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.742173 7822 ts_tablet_manager.cc:1428] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:33.742921 7454 heartbeater.cc:503] Master 127.0.98.60:34943 was elected leader, sending a full tablet report...
I20260929 18:31:33.743240 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.740241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:39595)
0929 18:31:33.743084 (+ 2843us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.743139 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2784,"thread_start_us":86,"threads_started":1}
I20260929 18:31:33.744640 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.743603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:35273)
0929 18:31:33.744519 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.744571 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":84,"threads_started":1}
I20260929 18:31:33.745105 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.738926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:59531)
0929 18:31:33.744955 (+ 6029us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.745018 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5950,"thread_start_us":82,"threads_started":1}
I20260929 18:31:33.745218 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.743344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:45987)
0929 18:31:33.745074 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.745128 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":79,"threads_started":1}
I20260929 18:31:33.745290 7727 heartbeater.cc:503] Master 127.0.98.60:34943 was elected leader, sending a full tablet report...
I20260929 18:31:33.745847 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.745482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:49191)
0929 18:31:33.745715 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.745773 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260929 18:31:33.749213 7594 tablet.cc:2369] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:33.768991 7826 raft_consensus.cc:482] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:33.769111 7826 raft_consensus.cc:504] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.770200 7826 leader_election.cc:288] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359), 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:33.774454 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.770542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39593 (local address 127.0.98.3:44111)
0929 18:31:33.770648 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.770653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.770670 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:31:33.770843 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.770846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.770897 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.770984 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.771421 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.772232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.773374 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.773827 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.773914 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.773990 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.774239 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.774324 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:31:33.774477 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.770529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58261 (local address 127.0.98.2:43359)
0929 18:31:33.770648 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.770653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.770671 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:31:33.770843 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.770846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.770897 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.770986 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:33.771421 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.772232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.773473 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.773897 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.774018 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.774076 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.774294 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.774374 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:31:33.774471 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.770458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44111 (local address 127.0.98.1:39593)
0929 18:31:33.770606 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.770622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.770725 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.771006 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.771009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.771032 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.771305 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.771311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.772388 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.772391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.773226 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.773235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.773971 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.773977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.774075 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.774131 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.774189 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.774274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":119,"spinlock_wait_cycles":6144}
I20260929 18:31:33.774469 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.770425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43359 (local address 127.0.98.1:58261)
0929 18:31:33.770606 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.770622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:33.770725 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:33.771006 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:33.771009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:33.771032 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:33.771304 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.771311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.772372 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.772375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.773319 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:33.773332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.774037 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.774040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:33.774143 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.774160 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:31:33.774255 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:31:33.774317 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":112,"spinlock_wait_cycles":16256}
I20260929 18:31:33.774887 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16f02f4c202451a893a274294a13abc" candidate_uuid: "c72cfe0c28df455e93bb2bf568e32887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a46a137b46439a93c69259ea1c4b05" is_pre_election: true
I20260929 18:31:33.775045 7681 raft_consensus.cc:2454] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72cfe0c28df455e93bb2bf568e32887 in term 0.
I20260929 18:31:33.775452 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16f02f4c202451a893a274294a13abc" candidate_uuid: "c72cfe0c28df455e93bb2bf568e32887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c6f48a2b024d48a5f3810649df84da" is_pre_election: true
I20260929 18:31:33.775466 7344 leader_election.cc:302] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a46a137b46439a93c69259ea1c4b05, c72cfe0c28df455e93bb2bf568e32887; no voters:
I20260929 18:31:33.775709 7545 raft_consensus.cc:2454] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c72cfe0c28df455e93bb2bf568e32887 in term 0.
I20260929 18:31:33.775748 7826 raft_consensus.cc:2781] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:33.775799 7826 raft_consensus.cc:482] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:33.775835 7826 raft_consensus.cc:3037] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:33.776903 7826 raft_consensus.cc:504] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.777226 7826 leader_election.cc:288] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [CANDIDATE]: Term 1 election: Requested vote from peers 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359), 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111)
I20260929 18:31:33.777529 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16f02f4c202451a893a274294a13abc" candidate_uuid: "c72cfe0c28df455e93bb2bf568e32887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c6f48a2b024d48a5f3810649df84da"
I20260929 18:31:33.777597 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16f02f4c202451a893a274294a13abc" candidate_uuid: "c72cfe0c28df455e93bb2bf568e32887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a46a137b46439a93c69259ea1c4b05"
I20260929 18:31:33.777648 7545 raft_consensus.cc:3037] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:33.777734 7681 raft_consensus.cc:3037] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:33.778614 7545 raft_consensus.cc:2454] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72cfe0c28df455e93bb2bf568e32887 in term 1.
I20260929 18:31:33.778695 7681 raft_consensus.cc:2454] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c72cfe0c28df455e93bb2bf568e32887 in term 1.
I20260929 18:31:33.778899 7343 leader_election.cc:302] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6f48a2b024d48a5f3810649df84da, c72cfe0c28df455e93bb2bf568e32887; no voters:
I20260929 18:31:33.779100 7826 raft_consensus.cc:2781] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:33.779318 7826 raft_consensus.cc:686] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 1 LEADER]: Becoming Leader. State: Replica: c72cfe0c28df455e93bb2bf568e32887, State: Running, Role: LEADER
I20260929 18:31:33.779582 7826 consensus_queue.cc:237] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } }
I20260929 18:31:33.779961 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.779754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:41363)
0929 18:31:33.779850 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.779904 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:31:33.781936 7274 catalog_manager.cc:5495] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 reported cstate change: term changed from 0 to 1, leader changed from <none> to c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c72cfe0c28df455e93bb2bf568e32887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72cfe0c28df455e93bb2bf568e32887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:33.796016 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.792140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.1:44643)
0929 18:31:33.792267 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:33.792274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:33.792349 (+ 75us) server_negotiation.cc:407] Connection header received
0929 18:31:33.792577 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:33.792580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:33.792636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:33.792758 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:33.793014 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.793572 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.794514 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:33.794756 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:33.794875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:33.795152 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:33.795604 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:33.795740 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:33.795809 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:31:33.795867 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:31:33.798692 7681 raft_consensus.cc:1261] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Refusing update from remote peer c72cfe0c28df455e93bb2bf568e32887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:33.799007 7545 raft_consensus.cc:1261] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Refusing update from remote peer c72cfe0c28df455e93bb2bf568e32887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:33.799244 7833 consensus_queue.cc:1035] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a46a137b46439a93c69259ea1c4b05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:33.799577 7826 consensus_queue.cc:1035] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c6f48a2b024d48a5f3810649df84da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:33.801286 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.800958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:33417)
0929 18:31:33.801136 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.801196 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 18:31:33.803292 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.802990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:35613)
0929 18:31:33.803152 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.803208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 18:31:33.806054 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.805774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.3:48767)
0929 18:31:33.805916 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.805969 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 18:31:33.806635 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.806354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.2:43781)
0929 18:31:33.806517 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.806570 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 18:31:33.808647 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.808367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:35923)
0929 18:31:33.808499 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.808552 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:31:33.808915 7839 mvcc.cc:205] Tried to move back new op lower bound from 7334734617794387968 to 7334734617724198912. Current Snapshot: MvccSnapshot[applied={T|T < 7334734617794387968 or (T in {7334734617794387968})}]
I20260929 18:31:33.809372 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.809143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35157 (local address 127.0.98.1:53373)
0929 18:31:33.809248 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.809305 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:31:33.814594 7273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:33.814756 7273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:33.818140 7273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1693b3cb4464c13877cc60f68e9d78f: Sending DeleteTablet for 3 replicas of tablet c16f02f4c202451a893a274294a13abc
I20260929 18:31:33.818868 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet c16f02f4c202451a893a274294a13abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43962
I20260929 18:31:33.818919 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet c16f02f4c202451a893a274294a13abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33802
I20260929 18:31:33.819208 7846 tablet_replica.cc:330] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887: stopping tablet replica
I20260929 18:31:33.819340 7846 raft_consensus.cc:2229] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:33.819340 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet c16f02f4c202451a893a274294a13abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59706
I20260929 18:31:33.819639 7846 raft_consensus.cc:2258] T c16f02f4c202451a893a274294a13abc P c72cfe0c28df455e93bb2bf568e32887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:33.819892 7847 tablet_replica.cc:330] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: stopping tablet replica
I20260929 18:31:33.819996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7237
I20260929 18:31:33.821401 7847 raft_consensus.cc:2229] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:33.821573 7847 raft_consensus.cc:2258] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:33.821933 7848 tablet_replica.cc:330] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: stopping tablet replica
I20260929 18:31:33.822058 7848 raft_consensus.cc:2229] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:33.822196 7848 raft_consensus.cc:2258] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:33.822381 7847 ts_tablet_manager.cc:1905] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:33.822875 7848 ts_tablet_manager.cc:1905] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:33.825033 7847 ts_tablet_manager.cc:1918] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:33.825125 7847 log.cc:1196] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/c16f02f4c202451a893a274294a13abc
I20260929 18:31:33.825214 7848 ts_tablet_manager.cc:1918] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:33.825284 7848 log.cc:1196] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/c16f02f4c202451a893a274294a13abc
I20260929 18:31:33.825435 7847 ts_tablet_manager.cc:1939] T c16f02f4c202451a893a274294a13abc P 01c6f48a2b024d48a5f3810649df84da: Deleting consensus metadata
I20260929 18:31:33.825549 7848 ts_tablet_manager.cc:1939] T c16f02f4c202451a893a274294a13abc P 45a46a137b46439a93c69259ea1c4b05: Deleting consensus metadata
I20260929 18:31:33.826156 7262 catalog_manager.cc:4841] TS 45a46a137b46439a93c69259ea1c4b05 (127.0.98.3:44111): tablet c16f02f4c202451a893a274294a13abc (table test-table [id=67a39ccdb4ee440fbaa2dd87e93a0994]) successfully deleted
I20260929 18:31:33.826218 7261 catalog_manager.cc:4841] TS 01c6f48a2b024d48a5f3810649df84da (127.0.98.2:43359): tablet c16f02f4c202451a893a274294a13abc (table test-table [id=67a39ccdb4ee440fbaa2dd87e93a0994]) successfully deleted
I20260929 18:31:33.827797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7462
I20260929 18:31:33.828064 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:33.827760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44643 (local address 127.0.0.1:43982)
0929 18:31:33.827908 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:33.827965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:31:33.828614 7259 catalog_manager.cc:4584] TS c72cfe0c28df455e93bb2bf568e32887 (127.0.98.1:44643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c16f02f4c202451a893a274294a13abc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44643: connect: Connection refused (error 111)
I20260929 18:31:33.835441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7598
I20260929 18:31:33.840728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 6954
I20260929 18:31:33.846326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7058
2026-09-29T18:31:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5246 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:33 dist-test-slave-fmv8 krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:33 dist-test-slave-fmv8 krb5kdc[7851](info): set up 2 sockets
Sep 29 18:31:33 dist-test-slave-fmv8 krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260929 18:31:34.884919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:34 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 18:31:35.003476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:35.115658 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:35.115978 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:35.116066 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:35.120203 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:35.120283 7864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:31:35.120316 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:35.120360 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:35.120398 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:35.120432 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:35.125643 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:35.126711 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:35.127812 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:35.136066 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.136096 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.136085 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:35.136516 7864 server_base.cc:1034] running on GCE node
I20260929 18:31:35.136991 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:35.138003 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:35.139139 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790706695139118 us; error 32 us; skew 500 ppm
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:35.141856 7864 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 18:31:35.142736 7864 webserver.cc:460] Webserver started at http://127.0.98.62:40309/ using document root <none> and password file <none>
I20260929 18:31:35.143257 7864 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:35.143316 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:35.143527 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:35.145490 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "b41c4ba959524f1fa81371acd612b072"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cff1287b4babcc322f53ae9b31bd65e9"
server_key_iv: "014c2d6234efbf2dc23e22ac5fafa322"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.146029 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "b41c4ba959524f1fa81371acd612b072"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cff1287b4babcc322f53ae9b31bd65e9"
server_key_iv: "014c2d6234efbf2dc23e22ac5fafa322"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.149538 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:35.151819 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.153065 7864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:35.153184 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "b41c4ba959524f1fa81371acd612b072"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cff1287b4babcc322f53ae9b31bd65e9"
server_key_iv: "014c2d6234efbf2dc23e22ac5fafa322"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.153282 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:35.184226 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:35.187600 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:35.187799 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:35.195613 7864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45023
I20260929 18:31:35.195637 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45023 every 8 connection(s)
I20260929 18:31:35.196540 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:35.199222 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:35.200078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7864
I20260929 18:31:35.200273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:35.200604 392 external_mini_cluster.cc:1428] Setting key e5db02516181e618057984b11b974fc3
I20260929 18:31:35.205914 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Bootstrap starting.
I20260929 18:31:35.208503 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:35.209208 7942 log.cc:824] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:35.211465 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: No bootstrap required, opened a new log
I20260929 18:31:35.215580 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } }
I20260929 18:31:35.215840 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:35.215895 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41c4ba959524f1fa81371acd612b072, State: Initialized, Role: FOLLOWER
I20260929 18:31:35.216138 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.202051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.0.98.62:45023)
0929 18:31:35.202473 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.202492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.202541 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:31:35.203338 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.203361 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.203772 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.204164 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.204858 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.206290 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.207140 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.207800 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.207961 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.210572 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.210577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.210593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.210626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.213192 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.213592 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.213599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.213605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.213739 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.213989 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.213992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.213994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.214507 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.214705 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.214897 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.215191 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260929 18:31:35.216377 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } }
I20260929 18:31:35.216526 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:35.216610 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:35.216706 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:35.217567 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } }
I20260929 18:31:35.217911 7942 leader_election.cc:302] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41c4ba959524f1fa81371acd612b072; no voters:
I20260929 18:31:35.218240 7942 leader_election.cc:288] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:35.218353 7947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:35.218544 7947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [term 1 LEADER]: Becoming Leader. State: Replica: b41c4ba959524f1fa81371acd612b072, State: Running, Role: LEADER
I20260929 18:31:35.218873 7947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } }
I20260929 18:31:35.219475 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:35.220598 7949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b41c4ba959524f1fa81371acd612b072. Latest consensus state: current_term: 1 leader_uuid: "b41c4ba959524f1fa81371acd612b072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } }
I20260929 18:31:35.220721 7949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:35.220988 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b41c4ba959524f1fa81371acd612b072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41c4ba959524f1fa81371acd612b072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45023 } } }
I20260929 18:31:35.221081 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:35.221172 7956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:35.224231 7956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:35.229521 7956 catalog_manager.cc:1347] Generated new cluster ID: fdf17b3c50904129a68c1712fbbc3e13
I20260929 18:31:35.229595 7956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:35.242835 7956 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:35.243762 7956 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:35.250730 7956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Generated new TSK 0
I20260929 18:31:35.251386 7956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 18:31:35.285365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45023
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:35.406032 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:35.406414 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:35.406487 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:35.412611 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:35.412686 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:35.412750 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:35.412813 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:35.420397 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:35.421514 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:35.422924 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:35.435823 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.435989 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.435871 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:35.441560 7968 server_base.cc:1034] running on GCE node
I20260929 18:31:35.442040 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:35.442785 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:35.443976 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790706695443909 us; error 85 us; skew 500 ppm
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:35.447436 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:31:35.448493 7968 webserver.cc:460] Webserver started at http://127.0.98.1:34469/ using document root <none> and password file <none>
I20260929 18:31:35.449156 7968 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:35.449277 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:35.449563 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:35.451553 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "1088a52a955e42e78f35e53c52090ab3"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cb20294e6356f5ba596bdacba140cccf"
server_key_iv: "9a5f98325f2f4aacc8f5a37f7fb35bf6"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.452100 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1088a52a955e42e78f35e53c52090ab3"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cb20294e6356f5ba596bdacba140cccf"
server_key_iv: "9a5f98325f2f4aacc8f5a37f7fb35bf6"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.455691 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:35.458123 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.459223 7968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:35.459349 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "1088a52a955e42e78f35e53c52090ab3"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "cb20294e6356f5ba596bdacba140cccf"
server_key_iv: "9a5f98325f2f4aacc8f5a37f7fb35bf6"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.459452 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:35.490154 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:35.493083 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:35.493292 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:35.494668 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:35.494761 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.494833 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:35.494877 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:35.505791 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34157
I20260929 18:31:35.505841 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34157 every 8 connection(s)
I20260929 18:31:35.506865 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:35.508445 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.495521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56797 (local address 127.0.98.62:45023)
0929 18:31:35.495692 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.495698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.496479 (+ 781us) server_negotiation.cc:407] Connection header received
0929 18:31:35.497404 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.497408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.497462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.497542 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.499024 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.499551 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.500924 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.501155 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.501267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.504330 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.504337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.504343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.504379 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.506088 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.506566 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.506570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.506572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.506656 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.507009 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.507012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.507014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.507250 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.507352 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.508211 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.508324 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:31:35.508967 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.495873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.1:56797)
0929 18:31:35.496321 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.496369 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.497202 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.497845 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.497855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.498301 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.498881 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.498895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.499675 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.499679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.500801 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.500811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.501305 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.501309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.501535 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.502146 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.502165 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.504109 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.506194 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.506198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.506208 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.506448 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.506745 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.506747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.506749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.506879 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.507401 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.507407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.507697 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.507883 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.508136 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260929 18:31:35.510272 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45023
I20260929 18:31:35.510468 8099 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:35.510936 8099 heartbeater.cc:511] Master 127.0.98.62:45023 requested a full tablet report, sending...
I20260929 18:31:35.512341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 7968
I20260929 18:31:35.512466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:35.512652 7894 ts_manager.cc:194] Registered new tserver with Master: 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157)
I20260929 18:31:35.512817 392 external_mini_cluster.cc:1428] Setting key e10a0364497cdf907341f0e18b6ae6e5
I20260929 18:31:35.513934 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56797
I20260929 18:31:35.520326 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.512229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.1:58979)
0929 18:31:35.512522 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.512538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.512671 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.512977 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.512979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.513188 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.513299 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.513306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.514276 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.514279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.515050 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.515060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.515684 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.515688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.515778 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.516399 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.516411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.516749 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.518670 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.518673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.518674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.518857 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.519163 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.519165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.519166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.519241 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.519718 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.519720 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.519796 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.519943 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.520077 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20260929 18:31:35.520370 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.512292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58979 (local address 127.0.98.62:45023)
0929 18:31:35.512534 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.512537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.512606 (+ 69us) server_negotiation.cc:407] Connection header received
0929 18:31:35.512776 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.512779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.512842 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.512926 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.513435 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.514138 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.515173 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.515557 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.515675 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.516898 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.516904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.516909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.516938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.518565 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.518977 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.518982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.518985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.519070 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.519369 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.519374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.519379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.519614 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.519720 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.520016 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.520140 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 18:31:35.539323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45023
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:35.645910 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:35.646142 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:35.646183 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:35.650035 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:35.650079 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:35.650115 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:35.650146 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:35.655191 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:35.656080 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:35.657317 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:35.666836 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.666853 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:35.666913 8108 server_base.cc:1034] running on GCE node
W20260929 18:31:35.666855 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:35.667490 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:35.668030 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:35.669204 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790706695669191 us; error 31 us; skew 500 ppm
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:35.671733 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:31:35.672621 8108 webserver.cc:460] Webserver started at http://127.0.98.2:43675/ using document root <none> and password file <none>
I20260929 18:31:35.673156 8108 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:35.673242 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:35.673465 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:35.675371 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "343a15104d92409a8f5249e4268e45b9"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "b788d0261120629a6e90c69e790bd504"
server_key_iv: "fdcde2b2b3fefa8e0e0467fc5b965ee3"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.675918 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "343a15104d92409a8f5249e4268e45b9"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "b788d0261120629a6e90c69e790bd504"
server_key_iv: "fdcde2b2b3fefa8e0e0467fc5b965ee3"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.679399 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:35.681707 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.682864 8108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:35.682992 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "343a15104d92409a8f5249e4268e45b9"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "b788d0261120629a6e90c69e790bd504"
server_key_iv: "fdcde2b2b3fefa8e0e0467fc5b965ee3"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.683094 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:35.717298 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:35.720417 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:35.720620 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:35.722105 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:35.722175 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.722240 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:35.722289 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:35.733112 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37865
I20260929 18:31:35.733263 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37865 every 8 connection(s)
I20260929 18:31:35.734236 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:35.735482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8108
I20260929 18:31:35.735605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:35.735911 392 external_mini_cluster.cc:1428] Setting key 9da2fa0c3b0a48b044baecb45321ff2e
I20260929 18:31:35.736967 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.722992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41631 (local address 127.0.98.62:45023)
0929 18:31:35.723115 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.723118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.724320 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 18:31:35.725530 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.725534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.725594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.725704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.727330 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.728238 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.729733 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.730078 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.730211 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.733014 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.733021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.733026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.733059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.734738 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.735289 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.735293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.735295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.735370 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.735753 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.735755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.735757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.735934 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.736055 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.736704 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.736815 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:31:35.737735 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.723566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.2:41631)
0929 18:31:35.724078 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.724121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.725283 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.725930 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.725939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.726433 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.727154 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.727178 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.728387 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.728390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.729596 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.729606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.730257 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.730260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.730433 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.731070 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.731089 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.732810 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.734864 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.734873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.734889 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.735198 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.735479 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.735483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.735485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.735662 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.736126 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.736130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.736450 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.736661 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.736967 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260929 18:31:35.739005 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45023
I20260929 18:31:35.739151 8239 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:35.739615 8239 heartbeater.cc:511] Master 127.0.98.62:45023 requested a full tablet report, sending...
I20260929 18:31:35.740664 7896 ts_manager.cc:194] Registered new tserver with Master: 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865)
I20260929 18:31:35.741493 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41631
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:35.747875 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.741236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.2:52163)
0929 18:31:35.741494 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.741506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.741613 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.741920 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.741922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.742075 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.742202 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.742207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.742919 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.742922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.743485 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.743492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.743919 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.743922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.744008 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.744527 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.744540 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.744855 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.746512 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.746515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.746516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.746776 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.747009 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.747011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.747012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.747082 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.747482 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.747485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.747560 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.747640 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.747737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260929 18:31:35.747882 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.741164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52163 (local address 127.0.98.62:45023)
0929 18:31:35.741270 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.741274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.741575 (+ 301us) server_negotiation.cc:407] Connection header received
0929 18:31:35.741699 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.741702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.741753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.741821 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.742314 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.742800 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.743595 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.743810 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.743916 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.744985 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.744988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.744991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.745019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.746410 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.746862 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.746865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.746866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.746932 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.747153 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.747155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.747156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.747382 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.747479 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.747699 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.747784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:31:35.763055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45023
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:35.872634 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:35.872881 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:35.872924 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:35.876924 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:35.876977 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:35.877015 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:31:35.877096 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:35.882205 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:35.883215 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:35.884289 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:35.893705 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.893764 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:35.893774 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:35.893918 8248 server_base.cc:1034] running on GCE node
I20260929 18:31:35.894373 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:35.894976 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:35.896196 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790706695896163 us; error 53 us; skew 500 ppm
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:35.898849 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:31:35.899676 8248 webserver.cc:460] Webserver started at http://127.0.98.3:42445/ using document root <none> and password file <none>
I20260929 18:31:35.900275 8248 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:35.900363 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:35.900571 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:35.902514 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "926b08b0e0994468afcd33c8220501df"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "4d2ba345ce1cc357c96f6ee2ffda3eff"
server_key_iv: "fcefbc07cfb2bc4bb9177c54a7420a54"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.903086 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "926b08b0e0994468afcd33c8220501df"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "4d2ba345ce1cc357c96f6ee2ffda3eff"
server_key_iv: "fcefbc07cfb2bc4bb9177c54a7420a54"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.906715 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:35.909214 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.910526 8248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:35.910662 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "926b08b0e0994468afcd33c8220501df"
format_stamp: "Formatted at 2026-09-29 18:31:35 on dist-test-slave-fmv8"
server_key: "4d2ba345ce1cc357c96f6ee2ffda3eff"
server_key_iv: "fcefbc07cfb2bc4bb9177c54a7420a54"
server_key_version: "encryptionkey@0"
I20260929 18:31:35.910775 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:35.923202 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:35.925908 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:35.926102 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:35.927452 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:35.927533 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:35.927608 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:35.927651 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:35.938021 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36035
I20260929 18:31:35.938088 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36035 every 8 connection(s)
I20260929 18:31:35.939020 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:35.941722 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.928259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53183 (local address 127.0.98.62:45023)
0929 18:31:35.928415 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.928422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.929201 (+ 779us) server_negotiation.cc:407] Connection header received
0929 18:31:35.930235 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.930239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.930291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.930363 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.932099 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.933009 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.934699 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.935099 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.935207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.937961 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.937965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.937969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.937998 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.939692 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.940313 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.940317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.940319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.940408 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.940697 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.940700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.940701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.940893 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.940966 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.941487 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.941586 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:31:35.942452 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.928612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.3:53183)
0929 18:31:35.929035 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.929082 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.929980 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.930811 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.930825 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.931325 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.931917 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.931939 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.933141 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.933145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.934565 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.934577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.935246 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.935249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.935384 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.935981 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.936008 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.937793 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.939819 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.939833 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.939863 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.940215 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.940496 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.940498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.940500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.940614 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.940977 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.940982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.941186 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.941386 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.941615 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260929 18:31:35.943755 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45023
I20260929 18:31:35.943903 8379 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:35.944455 8379 heartbeater.cc:511] Master 127.0.98.62:45023 requested a full tablet report, sending...
I20260929 18:31:35.945655 7894 ts_manager.cc:194] Registered new tserver with Master: 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:35.946344 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53183
I20260929 18:31:35.948978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8248
I20260929 18:31:35.949112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:35.949414 392 external_mini_cluster.cc:1428] Setting key 6701896fe436e97de34544c8d5f014d5
I20260929 18:31:35.951385 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:31:35.953466 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.945946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40793 (local address 127.0.98.62:45023)
0929 18:31:35.946061 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.946064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.946833 (+ 769us) server_negotiation.cc:407] Connection header received
0929 18:31:35.946946 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.946948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.946984 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.947049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.947517 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.947993 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.948913 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.949191 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.949292 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.950441 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.950445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.950448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.950476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.951881 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.952356 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.952360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.952361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.952427 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.952699 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.952703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.952704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.952861 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.952959 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.953212 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.953325 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:31:35.953478 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.945883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.98.3:40793)
0929 18:31:35.946739 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:35.946752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:35.946830 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:35.947090 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:35.947093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:35.947304 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:35.947410 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.947417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.948116 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.948120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.948794 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:35.948803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.949294 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.949297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:35.949391 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.949957 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:35.949974 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:35.950302 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:35.951990 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.951995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.951997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.952244 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.952516 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:35.952520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:35.952522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:35.952605 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:35.952961 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:35.952964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:35.953053 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:31:35.953152 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:31:35.953284 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":100,"threads_started":1}
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:31:35.974922 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:35.969140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.62:45023)
0929 18:31:35.969262 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.969267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.969280 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:35.969318 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.969320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.969355 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.969430 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.970114 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.970637 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.971428 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.971629 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.971706 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.974806 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34980: BlockingRecv error: recv got EOF from 127.0.0.1:34980 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 18:31:35 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:35.984802 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:35.976642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.98.62:45023)
0929 18:31:35.976785 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:35.976789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:35.976909 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:31:35.977045 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:35.977048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:35.977093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:35.977172 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:35.977723 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.978212 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.979010 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:35.979224 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:35.979345 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:35.981595 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:35.981598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:35.981600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:35.981620 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:35.983012 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.983420 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.983423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.983427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.983534 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:35.983889 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:35.983893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:35.983895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:35.984135 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:35.984282 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:35.984510 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:31:35.984618 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:31:35.988102 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:35.991174 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:36.006570 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.000375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.98.3:36035)
0929 18:31:36.001326 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.001340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.001364 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:36.001436 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.001441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.001655 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.001822 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.002261 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003474 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.004952 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.005782 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.005912 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.006084 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.006318 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.006421 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":114,"threads_started":1}
I20260929 18:31:36.006623 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.000243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34157 (local address 127.0.0.1:60282)
0929 18:31:36.000790 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.000803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.000883 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.002081 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.002088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.002111 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.002258 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.002265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003588 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.003591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.004987 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.004997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.005956 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.005959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.006101 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.006160 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.006274 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.006440 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":200,"thread_start_us":94,"threads_started":1}
I20260929 18:31:36.006623 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.000216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36035 (local address 127.0.0.1:49676)
0929 18:31:36.000693 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.000750 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.000883 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.001992 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.002001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.002028 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.002156 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.002164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003628 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.003631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.004831 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.004839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.005938 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.005942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.006106 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.006160 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.006274 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.006440 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":128,"thread_start_us":110,"threads_started":1}
I20260929 18:31:36.007133 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.000964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.0.98.1:34157)
0929 18:31:36.001630 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.001645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.001666 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:31:36.001741 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.001749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.001933 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.002079 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.002382 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003475 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.005114 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.005858 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.005950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.006086 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.006877 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.006973 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":93,"threads_started":1}
I20260929 18:31:36.007844 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.000976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.0.1:54626)
0929 18:31:36.001343 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.001357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.001440 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.002177 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.002180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.002192 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.002302 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.002310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003358 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.003364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.004987 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.004997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.007436 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.007440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.007523 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.007554 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.007637 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.007705 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":150,"thread_start_us":91,"threads_started":1}
I20260929 18:31:36.007915 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.001234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.98.2:37865)
0929 18:31:36.001720 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.001733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.001757 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:31:36.001835 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.001841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.002012 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.002433 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.002542 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.003243 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.006154 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.007175 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.007405 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.007572 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.007667 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.007772 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":85,"threads_started":1}
I20260929 18:31:36.009984 8313 tablet_service.cc:1467] Processing CreateTablet for tablet e86c9989603a47bda5625786406b062a (DEFAULT_TABLE table=test-table [id=37592c09cf95490db26c72ccbc3cdebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.010588 8173 tablet_service.cc:1467] Processing CreateTablet for tablet e86c9989603a47bda5625786406b062a (DEFAULT_TABLE table=test-table [id=37592c09cf95490db26c72ccbc3cdebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.011057 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86c9989603a47bda5625786406b062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.011534 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86c9989603a47bda5625786406b062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.017889 8408 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Bootstrap starting.
I20260929 18:31:36.017891 8407 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Bootstrap starting.
I20260929 18:31:36.020193 8408 tablet_bootstrap.cc:654] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.020372 8407 tablet_bootstrap.cc:654] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.021014 8408 log.cc:824] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:36.021123 8407 log.cc:824] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:36.020972 8033 tablet_service.cc:1467] Processing CreateTablet for tablet e86c9989603a47bda5625786406b062a (DEFAULT_TABLE table=test-table [id=37592c09cf95490db26c72ccbc3cdebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.022006 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86c9989603a47bda5625786406b062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.023030 8408 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: No bootstrap required, opened a new log
I20260929 18:31:36.023240 8408 ts_tablet_manager.cc:1397] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 18:31:36.026047 8408 raft_consensus.cc:348] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.026284 8408 raft_consensus.cc:374] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.026341 8408 raft_consensus.cc:729] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926b08b0e0994468afcd33c8220501df, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.026753 8408 consensus_queue.cc:260] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.027355 8379 heartbeater.cc:503] Master 127.0.98.62:45023 was elected leader, sending a full tablet report...
I20260929 18:31:36.027637 8407 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: No bootstrap required, opened a new log
I20260929 18:31:36.027758 8408 ts_tablet_manager.cc:1428] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260929 18:31:36.027801 8407 ts_tablet_manager.cc:1397] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260929 18:31:36.028904 8412 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Bootstrap starting.
I20260929 18:31:36.030738 8412 tablet_bootstrap.cc:654] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.031541 8412 log.cc:824] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:36.031646 8407 raft_consensus.cc:348] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.031898 8407 raft_consensus.cc:374] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.031957 8407 raft_consensus.cc:729] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.032397 8407 consensus_queue.cc:260] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.033020 8239 heartbeater.cc:503] Master 127.0.98.62:45023 was elected leader, sending a full tablet report...
I20260929 18:31:36.033257 8407 ts_tablet_manager.cc:1428] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:31:36.033455 8412 tablet_bootstrap.cc:492] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: No bootstrap required, opened a new log
I20260929 18:31:36.033636 8412 ts_tablet_manager.cc:1397] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:31:36.036633 8412 raft_consensus.cc:348] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.036906 8412 raft_consensus.cc:374] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.036964 8412 raft_consensus.cc:729] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.037350 8412 consensus_queue.cc:260] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.037953 8099 heartbeater.cc:503] Master 127.0.98.62:45023 was elected leader, sending a full tablet report...
I20260929 18:31:36.038256 8412 ts_tablet_manager.cc:1428] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:36.147974 8414 raft_consensus.cc:482] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:36.148268 8414 raft_consensus.cc:504] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.149518 8414 leader_election.cc:288] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035), 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157)
I20260929 18:31:36.154016 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.149712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36035 (local address 127.0.98.2:54285)
0929 18:31:36.149931 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.149946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.150036 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.150308 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.150312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.150349 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.150485 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.150490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.151510 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.151513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.152955 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.152966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.153693 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.153696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.153774 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.153793 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.153852 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.153910 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"spinlock_wait_cycles":5376}
I20260929 18:31:36.154105 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.149837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54285 (local address 127.0.98.3:36035)
0929 18:31:36.149967 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.149973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.150001 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:31:36.150160 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.150163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.150206 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.150270 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.150591 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.151380 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.153071 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.153580 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.153686 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.153749 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.153887 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.153957 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:36.154820 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.149935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41753 (local address 127.0.98.1:34157)
0929 18:31:36.150064 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.150068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.150340 (+ 272us) server_negotiation.cc:407] Connection header received
0929 18:31:36.150436 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.150439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.150487 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.150570 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.150822 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.151526 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.153390 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.153979 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.154078 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.154135 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.154666 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.154728 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:36.154742 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86c9989603a47bda5625786406b062a" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df" is_pre_election: true
I20260929 18:31:36.154847 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.149780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34157 (local address 127.0.98.2:41753)
0929 18:31:36.150204 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.150214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.150350 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.150599 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.150601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.150624 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.150709 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.150713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.151652 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.151655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.153277 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.153285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.154387 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.154392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.154530 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.154553 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.154635 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.154696 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":26752,"thread_start_us":136,"threads_started":1}
I20260929 18:31:36.155114 8333 raft_consensus.cc:2454] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 0.
I20260929 18:31:36.155438 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86c9989603a47bda5625786406b062a" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1088a52a955e42e78f35e53c52090ab3" is_pre_election: true
I20260929 18:31:36.155674 8125 leader_election.cc:302] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a15104d92409a8f5249e4268e45b9, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:36.155763 8053 raft_consensus.cc:2454] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 0.
I20260929 18:31:36.155959 8414 raft_consensus.cc:2781] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:36.156085 8414 raft_consensus.cc:482] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:36.156167 8414 raft_consensus.cc:3037] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.157291 8414 raft_consensus.cc:504] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.157656 8414 leader_election.cc:288] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 election: Requested vote from peers 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035), 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157)
I20260929 18:31:36.157982 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86c9989603a47bda5625786406b062a" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df"
I20260929 18:31:36.158015 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86c9989603a47bda5625786406b062a" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1088a52a955e42e78f35e53c52090ab3"
I20260929 18:31:36.158111 8333 raft_consensus.cc:3037] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.158149 8053 raft_consensus.cc:3037] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.159230 8053 raft_consensus.cc:2454] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 1.
I20260929 18:31:36.159399 8333 raft_consensus.cc:2454] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 1.
I20260929 18:31:36.159559 8126 leader_election.cc:302] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 343a15104d92409a8f5249e4268e45b9; no voters:
I20260929 18:31:36.159762 8414 raft_consensus.cc:2781] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:36.159981 8414 raft_consensus.cc:686] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 1 LEADER]: Becoming Leader. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Running, Role: LEADER
I20260929 18:31:36.160352 8414 consensus_queue.cc:237] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260929 18:31:36.163599 7894 catalog_manager.cc:5495] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a15104d92409a8f5249e4268e45b9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: HEALTHY } } }
W20260929 18:31:36.189999 8380 tablet.cc:2369] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:36.194212 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.190741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.2:37865)
0929 18:31:36.190877 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.190880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.190925 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:31:36.191082 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.191084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.191139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.191225 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:36.191410 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.191915 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.192814 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.193033 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.193110 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.193363 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.193799 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.193885 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.193968 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.194016 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:31:36.200502 8053 raft_consensus.cc:1261] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Refusing update from remote peer 343a15104d92409a8f5249e4268e45b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:36.200635 8333 raft_consensus.cc:1261] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Refusing update from remote peer 343a15104d92409a8f5249e4268e45b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:36.201200 8414 consensus_queue.cc:1035] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.201367 8417 consensus_queue.cc:1035] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.209450 8420 mvcc.cc:205] Tried to move back new op lower bound from 7334734627630014464 to 7334734627475484672. Current Snapshot: MvccSnapshot[applied={T|T < 7334734627630014464}]
I20260929 18:31:36.218339 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:36.218485 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:36.220808 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Sending DeleteTablet for 3 replicas of tablet e86c9989603a47bda5625786406b062a
I20260929 18:31:36.221593 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet e86c9989603a47bda5625786406b062a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260929 18:31:36.221621 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet e86c9989603a47bda5625786406b062a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60282
I20260929 18:31:36.221689 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet e86c9989603a47bda5625786406b062a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54626
I20260929 18:31:36.222147 8432 tablet_replica.cc:330] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: stopping tablet replica
I20260929 18:31:36.222244 8431 tablet_replica.cc:330] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: stopping tablet replica
I20260929 18:31:36.222450 8432 raft_consensus.cc:2229] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:36.222568 8431 raft_consensus.cc:2229] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:36.222832 8432 raft_consensus.cc:2258] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.222882 8431 raft_consensus.cc:2258] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.222959 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:31:36.223368 8430 tablet_replica.cc:330] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: stopping tablet replica
W20260929 18:31:36.223361 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:36.224395 8432 ts_tablet_manager.cc:1905] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.224593 8431 ts_tablet_manager.cc:1905] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.227173 8432 ts_tablet_manager.cc:1918] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:36.227327 8432 log.cc:1196] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/e86c9989603a47bda5625786406b062a
I20260929 18:31:36.227561 8430 raft_consensus.cc:2229] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:36.227655 8432 ts_tablet_manager.cc:1939] T e86c9989603a47bda5625786406b062a P 343a15104d92409a8f5249e4268e45b9: Deleting consensus metadata
I20260929 18:31:36.227910 8430 raft_consensus.cc:2258] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.228960 7882 catalog_manager.cc:4841] TS 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865): tablet e86c9989603a47bda5625786406b062a (table test-table [id=37592c09cf95490db26c72ccbc3cdebf]) successfully deleted
I20260929 18:31:36.229130 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 (DEFAULT_TABLE table=test-table [id=7dbb95cee8694a638a989143e159d41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.229398 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c94a28c8b5f4de8a981e12a43495975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.229398 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 (DEFAULT_TABLE table=test-table [id=7dbb95cee8694a638a989143e159d41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.229586 8430 ts_tablet_manager.cc:1905] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.229614 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c94a28c8b5f4de8a981e12a43495975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.231726 8407 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Bootstrap starting.
I20260929 18:31:36.231726 8408 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Bootstrap starting.
I20260929 18:31:36.227216 8431 ts_tablet_manager.cc:1918] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:36.231901 8431 log.cc:1196] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/e86c9989603a47bda5625786406b062a
I20260929 18:31:36.232264 8431 ts_tablet_manager.cc:1939] T e86c9989603a47bda5625786406b062a P 1088a52a955e42e78f35e53c52090ab3: Deleting consensus metadata
I20260929 18:31:36.232702 8407 tablet_bootstrap.cc:654] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.232926 8408 tablet_bootstrap.cc:654] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.233702 8430 ts_tablet_manager.cc:1918] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:36.233762 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 (DEFAULT_TABLE table=test-table [id=7dbb95cee8694a638a989143e159d41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.233855 8430 log.cc:1196] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/e86c9989603a47bda5625786406b062a
I20260929 18:31:36.234019 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c94a28c8b5f4de8a981e12a43495975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.234217 8430 ts_tablet_manager.cc:1939] T e86c9989603a47bda5625786406b062a P 926b08b0e0994468afcd33c8220501df: Deleting consensus metadata
I20260929 18:31:36.234323 8408 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: No bootstrap required, opened a new log
I20260929 18:31:36.234406 8408 ts_tablet_manager.cc:1397] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:31:36.234826 8408 raft_consensus.cc:348] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.234949 8408 raft_consensus.cc:374] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.234994 8408 raft_consensus.cc:729] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926b08b0e0994468afcd33c8220501df, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.235117 8408 consensus_queue.cc:260] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.235414 8408 ts_tablet_manager.cc:1428] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:36.235476 7881 catalog_manager.cc:4841] TS 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035): tablet e86c9989603a47bda5625786406b062a (table test-table [id=37592c09cf95490db26c72ccbc3cdebf]) successfully deleted
I20260929 18:31:36.236231 8412 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Bootstrap starting.
I20260929 18:31:36.236439 7882 catalog_manager.cc:4841] TS 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157): tablet e86c9989603a47bda5625786406b062a (table test-table [id=37592c09cf95490db26c72ccbc3cdebf]) successfully deleted
I20260929 18:31:36.237075 8407 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: No bootstrap required, opened a new log
I20260929 18:31:36.237156 8412 tablet_bootstrap.cc:654] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.237164 8407 ts_tablet_manager.cc:1397] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260929 18:31:36.237579 8407 raft_consensus.cc:348] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.237689 8407 raft_consensus.cc:374] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.237725 8407 raft_consensus.cc:729] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.237833 8407 consensus_queue.cc:260] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.238065 8407 ts_tablet_manager.cc:1428] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:31:36.238901 8412 tablet_bootstrap.cc:492] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: No bootstrap required, opened a new log
I20260929 18:31:36.238981 8412 ts_tablet_manager.cc:1397] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:31:36.239374 8412 raft_consensus.cc:348] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.239480 8412 raft_consensus.cc:374] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.239511 8412 raft_consensus.cc:729] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.239612 8412 consensus_queue.cc:260] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.239836 8412 ts_tablet_manager.cc:1428] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260929 18:31:36.258842 8100 tablet.cc:2369] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:36.323678 8415 raft_consensus.cc:482] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:36.323865 8415 raft_consensus.cc:504] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.325130 8415 leader_election.cc:288] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:36.329391 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.325484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55613 (local address 127.0.98.3:36035)
0929 18:31:36.325672 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.325677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.325794 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:31:36.325942 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.325945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.326003 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.326104 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.326391 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.327174 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.328347 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.328781 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.328884 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.328945 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.329179 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.329247 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:31:36.329393 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.325366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36035 (local address 127.0.98.1:55613)
0929 18:31:36.325687 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.325704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.325825 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.326118 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.326121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.326141 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.326268 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.326273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.327300 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.327306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.328221 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.328231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.328930 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.328934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.329040 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.329067 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.329155 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.329225 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":19,"thread_start_us":79,"threads_started":1}
I20260929 18:31:36.329986 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.325546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59197 (local address 127.0.98.2:37865)
0929 18:31:36.325668 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.325673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.326168 (+ 495us) server_negotiation.cc:407] Connection header received
0929 18:31:36.326364 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.326368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.326424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.326509 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.326766 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.327258 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.328739 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.329301 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.329397 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.329469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.329804 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.329888 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:36.329945 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c94a28c8b5f4de8a981e12a43495975" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df" is_pre_election: true
I20260929 18:31:36.330099 8333 raft_consensus.cc:2454] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:36.330425 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.325366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.98.1:59197)
0929 18:31:36.326082 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.326096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.326238 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.326520 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.326524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.326539 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.326659 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.326664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.327390 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.327394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.328594 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.328605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.329528 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.329532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.329643 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.329678 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.330194 (+ 516us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.330273 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":317,"thread_start_us":166,"threads_started":1}
I20260929 18:31:36.330475 7985 leader_election.cc:302] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:36.330744 8415 raft_consensus.cc:2781] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:36.330811 8415 raft_consensus.cc:482] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:36.330856 8415 raft_consensus.cc:3037] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.330958 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c94a28c8b5f4de8a981e12a43495975" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9" is_pre_election: true
I20260929 18:31:36.331173 8185 raft_consensus.cc:2454] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:36.331883 8415 raft_consensus.cc:504] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.332206 8415 leader_election.cc:288] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:36.332476 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c94a28c8b5f4de8a981e12a43495975" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9"
I20260929 18:31:36.332597 8185 raft_consensus.cc:3037] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.332598 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c94a28c8b5f4de8a981e12a43495975" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df"
I20260929 18:31:36.332710 8333 raft_consensus.cc:3037] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.333621 8333 raft_consensus.cc:2454] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:36.333799 8185 raft_consensus.cc:2454] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:36.333906 7985 leader_election.cc:302] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:36.334100 8415 raft_consensus.cc:2781] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:36.334328 8415 raft_consensus.cc:686] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Running, Role: LEADER
I20260929 18:31:36.334559 8415 consensus_queue.cc:237] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.336776 7894 catalog_manager.cc:5495] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1088a52a955e42e78f35e53c52090ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:31:36 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:36.389858 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.381729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45023 (local address 127.0.0.1:35000)
0929 18:31:36.382011 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.382027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.382134 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.382358 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.382360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.382723 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:36.382826 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.382833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.383589 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.383591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.384143 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.384180 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.384603 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.384606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.384674 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.385276 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:36.385289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:36.386846 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:36.388485 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:36.388493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:36.388498 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:36.388728 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:36.388969 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:36.388971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:36.388972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:36.389049 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:36.389389 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:36.389394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:36.389492 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.389584 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.389704 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260929 18:31:36.389882 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.381866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.98.62:45023)
0929 18:31:36.381996 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.382000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.382114 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:31:36.382205 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.382207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.382245 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.382325 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:36.382963 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.383489 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.384300 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.384503 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.384588 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.386965 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:36.386968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:36.386970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:36.386990 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:36.388369 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:36.388817 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:36.388820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:36.388821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:36.388889 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:36.389123 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:36.389125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:36.389126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:36.389296 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:36.389370 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.389634 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.389762 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":53}
I20260929 18:31:36.393600 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35000:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:31:36.394330 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:36.398839 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 323ab24569644368854154d8402f9ed0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=217b99af2229412180a30d66498a7034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:36.398890 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 323ab24569644368854154d8402f9ed0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=217b99af2229412180a30d66498a7034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:36.398998 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 323ab24569644368854154d8402f9ed0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=217b99af2229412180a30d66498a7034]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:31:36.399260 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323ab24569644368854154d8402f9ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.399288 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323ab24569644368854154d8402f9ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.399290 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323ab24569644368854154d8402f9ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.401916 8412 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: Bootstrap starting.
I20260929 18:31:36.401914 8407 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: Bootstrap starting.
I20260929 18:31:36.401948 8408 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: Bootstrap starting.
I20260929 18:31:36.402707 8412 tablet_bootstrap.cc:654] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.402804 8407 tablet_bootstrap.cc:654] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.402812 8408 tablet_bootstrap.cc:654] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.404062 8412 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: No bootstrap required, opened a new log
I20260929 18:31:36.404095 8407 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: No bootstrap required, opened a new log
I20260929 18:31:36.404135 8408 tablet_bootstrap.cc:492] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: No bootstrap required, opened a new log
I20260929 18:31:36.404175 8412 ts_tablet_manager.cc:1397] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:36.404210 8407 ts_tablet_manager.cc:1397] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:31:36.404225 8408 ts_tablet_manager.cc:1397] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:36.404618 8412 raft_consensus.cc:348] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.404604 8407 raft_consensus.cc:348] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.404604 8408 raft_consensus.cc:348] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.404733 8407 raft_consensus.cc:374] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.404733 8412 raft_consensus.cc:374] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.404733 8408 raft_consensus.cc:374] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.404778 8412 raft_consensus.cc:729] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.404779 8407 raft_consensus.cc:729] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.404786 8408 raft_consensus.cc:729] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926b08b0e0994468afcd33c8220501df, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.404901 8408 consensus_queue.cc:260] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.404901 8407 consensus_queue.cc:260] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.404911 8412 consensus_queue.cc:260] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.405154 8407 ts_tablet_manager.cc:1428] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:36.405154 8408 ts_tablet_manager.cc:1428] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:31:36.405184 8412 ts_tablet_manager.cc:1428] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:36.405169 8425 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.405313 8425 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.405375 8411 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.405380 8415 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.405479 8411 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.405483 8415 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 18:31:36.485447 8240 tablet.cc:2369] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:31:36.620908 8411 raft_consensus.cc:482] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:36.621099 8411 raft_consensus.cc:504] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.621402 8425 raft_consensus.cc:482] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:36.621511 8425 raft_consensus.cc:504] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.621933 8425 leader_election.cc:288] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:36.622403 8411 leader_election.cc:288] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157), 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865)
I20260929 18:31:36.622452 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df" is_pre_election: true
I20260929 18:31:36.622452 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1088a52a955e42e78f35e53c52090ab3" is_pre_election: true
I20260929 18:31:36.622601 8333 raft_consensus.cc:2454] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 0.
I20260929 18:31:36.622601 8053 raft_consensus.cc:2454] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 0.
I20260929 18:31:36.622961 8126 leader_election.cc:302] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 343a15104d92409a8f5249e4268e45b9; no voters:
I20260929 18:31:36.623268 8425 raft_consensus.cc:2781] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:36.623338 8425 raft_consensus.cc:482] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:36.623391 8425 raft_consensus.cc:3037] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.624706 8425 raft_consensus.cc:504] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.625106 8425 leader_election.cc:288] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 election: Requested vote from peers 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:36.625383 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1088a52a955e42e78f35e53c52090ab3"
I20260929 18:31:36.625494 8053 raft_consensus.cc:3037] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.625469 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "343a15104d92409a8f5249e4268e45b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df"
I20260929 18:31:36.625566 8333 raft_consensus.cc:3037] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:36.626538 8053 raft_consensus.cc:2454] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 1.
I20260929 18:31:36.626537 8333 raft_consensus.cc:2454] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a15104d92409a8f5249e4268e45b9 in term 1.
I20260929 18:31:36.626849 8126 leader_election.cc:302] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 343a15104d92409a8f5249e4268e45b9, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:36.626994 8425 raft_consensus.cc:2781] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:36.627100 8425 raft_consensus.cc:686] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [term 1 LEADER]: Becoming Leader. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Running, Role: LEADER
I20260929 18:31:36.627233 8425 consensus_queue.cc:237] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.627511 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.623179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38367 (local address 127.0.98.2:37865)
0929 18:31:36.623319 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.623325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.623450 (+ 125us) server_negotiation.cc:407] Connection header received
0929 18:31:36.623628 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.623632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.623680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.623759 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.624025 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.624736 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626150 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.626737 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626799 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.626861 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.627309 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.627385 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:31:36.627511 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.622787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53983 (local address 127.0.98.1:34157)
0929 18:31:36.623013 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.623018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.623032 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:36.623077 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.623080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.623139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.623216 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:31:36.623765 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.624434 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626052 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.626665 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626744 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.626815 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.627322 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.627388 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 18:31:36.627890 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.622942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.98.3:38367)
0929 18:31:36.623373 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.623388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.623509 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.623799 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.623802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.623820 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.623910 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.623915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.624861 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.624866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.626030 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.626040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.626946 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.627082 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.627157 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.627643 (+ 486us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.627742 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260929 18:31:36.627902 8417 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: TxnStatusTablet state changed. Reason: New leader 343a15104d92409a8f5249e4268e45b9. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.627992 8417 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:36.627915 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.622560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34157 (local address 127.0.98.3:53983)
0929 18:31:36.622830 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.622848 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.622941 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.623507 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.623510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.623523 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:31:36.623648 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.623654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.624541 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.624545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.625912 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.625923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.626953 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.626956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.627179 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.627213 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.627738 (+ 525us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.627795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":127,"thread_start_us":110,"threads_started":1}
I20260929 18:31:36.628304 8452 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:31:36.628489 8452 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:31:36.628820 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "926b08b0e0994468afcd33c8220501df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9" is_pre_election: true
I20260929 18:31:36.628820 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ab24569644368854154d8402f9ed0" candidate_uuid: "926b08b0e0994468afcd33c8220501df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1088a52a955e42e78f35e53c52090ab3" is_pre_election: true
I20260929 18:31:36.628970 8053 raft_consensus.cc:2379] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 926b08b0e0994468afcd33c8220501df in current term 1: Already voted for candidate 343a15104d92409a8f5249e4268e45b9 in this term.
I20260929 18:31:36.629204 7896 catalog_manager.cc:5495] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a15104d92409a8f5249e4268e45b9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:36.629456 8266 leader_election.cc:302] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 926b08b0e0994468afcd33c8220501df; no voters: 1088a52a955e42e78f35e53c52090ab3, 343a15104d92409a8f5249e4268e45b9
I20260929 18:31:36.630031 8411 raft_consensus.cc:2726] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:31:36.696211 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.692870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.2:37865)
0929 18:31:36.692987 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.692992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.693007 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:31:36.693125 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.693129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.693183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.693266 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:36.693496 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.694271 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.695075 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.695409 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.695507 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.695767 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.695844 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.695913 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.696013 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.696061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:36.696233 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.692748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.0.1:54644)
0929 18:31:36.692889 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.692905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.693009 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.693293 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.693295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.693305 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:36.693390 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.693394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.694363 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.694366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.694964 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.694972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.695516 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.695518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.695582 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.695623 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.695922 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.695932 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.695980 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.696047 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:31:36.698802 8053 raft_consensus.cc:1261] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Refusing update from remote peer 343a15104d92409a8f5249e4268e45b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:36.698823 8333 raft_consensus.cc:1261] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Refusing update from remote peer 343a15104d92409a8f5249e4268e45b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:36.699273 8425 consensus_queue.cc:1035] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.699487 8417 consensus_queue.cc:1035] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.700753 8415 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: TxnStatusTablet state changed. Reason: New leader 343a15104d92409a8f5249e4268e45b9. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.700870 8415 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.701402 8411 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: TxnStatusTablet state changed. Reason: New leader 343a15104d92409a8f5249e4268e45b9. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.701515 8411 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.702939 8425 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.703043 8425 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:36.703696 8421 mvcc.cc:205] Tried to move back new op lower bound from 7334734629674307584 to 7334734629386702848. Current Snapshot: MvccSnapshot[applied={T|T < 7334734629674307584 or (T in {7334734629674307584})}]
I20260929 18:31:36.703899 8415 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.704005 8415 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 1088a52a955e42e78f35e53c52090ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.705808 8414 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.705914 8414 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 343a15104d92409a8f5249e4268e45b9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:31:36.707182 8411 tablet_replica.cc:439] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "343a15104d92409a8f5249e4268e45b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } } }
I20260929 18:31:36.707293 8411 tablet_replica.cc:442] T 323ab24569644368854154d8402f9ed0 P 926b08b0e0994468afcd33c8220501df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:31:36.713794 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.708128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.98.1:34157)
0929 18:31:36.708272 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.708277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.709996 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 18:31:36.710231 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.710235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.710301 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.710434 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:36.710759 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.711440 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.712364 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.712596 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.712708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.712961 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.713386 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.713511 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.713572 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.713647 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:36.722121 8415 consensus_queue.cc:1035] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.725812 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.720563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52821 (local address 127.0.98.2:37865)
0929 18:31:36.720769 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.720774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.721358 (+ 584us) server_negotiation.cc:407] Connection header received
0929 18:31:36.721457 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.721460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.721507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.721587 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:36.721863 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.722564 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.724371 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.724675 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.724747 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.725224 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.725308 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.725353 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.725517 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.725661 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 18:31:36.726240 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.720438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.98.1:52821)
0929 18:31:36.721272 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.721287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.721382 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.721607 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.721610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.721617 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:36.721755 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.721760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.722665 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.722668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.724224 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.724240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.724855 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.724858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.724964 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.725012 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.725414 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.725419 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.725970 (+ 551us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.726057 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":98,"threads_started":1}
I20260929 18:31:36.728505 8415 consensus_queue.cc:1035] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:36.758404 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.754238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33661 (local address 127.0.98.1:34157)
0929 18:31:36.754348 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.754352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.754551 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:31:36.754647 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.754650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.754700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.754794 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:36.755089 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.755864 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.757205 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.757546 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.757652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.757956 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.758032 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.758099 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.758236 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.758284 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:31:36.758960 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.754155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34157 (local address 127.0.98.2:33661)
0929 18:31:36.754460 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:36.754473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:36.754568 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:36.754810 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:36.754813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:36.754821 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:31:36.754970 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.754977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.755990 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.755993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.757045 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:36.757054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.757677 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.757680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:36.757778 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.757823 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:31:36.758120 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:31:36.758126 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:31:36.758721 (+ 595us) client_negotiation.cc:240] Negotiation successful
0929 18:31:36.758831 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
I20260929 18:31:36.781945 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:36.782120 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:36.784016 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Sending DeleteTablet for 3 replicas of tablet 5c94a28c8b5f4de8a981e12a43495975
I20260929 18:31:36.784663 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260929 18:31:36.784683 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60282
I20260929 18:31:36.784994 8471 tablet_replica.cc:330] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: stopping tablet replica
I20260929 18:31:36.785117 8471 raft_consensus.cc:2229] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:36.785175 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c94a28c8b5f4de8a981e12a43495975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54626
I20260929 18:31:36.785240 8471 raft_consensus.cc:2258] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.785463 8472 tablet_replica.cc:330] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: stopping tablet replica
I20260929 18:31:36.785583 8472 raft_consensus.cc:2229] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:36.785637 8473 tablet_replica.cc:330] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: stopping tablet replica
I20260929 18:31:36.785743 8473 raft_consensus.cc:2229] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:36.785837 8473 raft_consensus.cc:2258] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.785851 8472 raft_consensus.cc:2258] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:36.786384 8471 ts_tablet_manager.cc:1905] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.786417 8473 ts_tablet_manager.cc:1905] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.789110 8473 ts_tablet_manager.cc:1918] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:36.789109 8471 ts_tablet_manager.cc:1918] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:36.789191 8471 log.cc:1196] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/5c94a28c8b5f4de8a981e12a43495975
I20260929 18:31:36.789191 8473 log.cc:1196] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/5c94a28c8b5f4de8a981e12a43495975
I20260929 18:31:36.789467 8473 ts_tablet_manager.cc:1939] T 5c94a28c8b5f4de8a981e12a43495975 P 343a15104d92409a8f5249e4268e45b9: Deleting consensus metadata
I20260929 18:31:36.789467 8471 ts_tablet_manager.cc:1939] T 5c94a28c8b5f4de8a981e12a43495975 P 926b08b0e0994468afcd33c8220501df: Deleting consensus metadata
I20260929 18:31:36.790162 8472 ts_tablet_manager.cc:1905] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:36.790234 7882 catalog_manager.cc:4841] TS 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865): tablet 5c94a28c8b5f4de8a981e12a43495975 (table test-table [id=7dbb95cee8694a638a989143e159d41b]) successfully deleted
I20260929 18:31:36.790905 7881 catalog_manager.cc:4841] TS 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035): tablet 5c94a28c8b5f4de8a981e12a43495975 (table test-table [id=7dbb95cee8694a638a989143e159d41b]) successfully deleted
I20260929 18:31:36.793467 8472 ts_tablet_manager.cc:1918] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:36.793545 8472 log.cc:1196] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/5c94a28c8b5f4de8a981e12a43495975
I20260929 18:31:36.793753 8472 ts_tablet_manager.cc:1939] T 5c94a28c8b5f4de8a981e12a43495975 P 1088a52a955e42e78f35e53c52090ab3: Deleting consensus metadata
I20260929 18:31:36.794330 7882 catalog_manager.cc:4841] TS 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157): tablet 5c94a28c8b5f4de8a981e12a43495975 (table test-table [id=7dbb95cee8694a638a989143e159d41b]) successfully deleted
Sep 29 18:31:36 dist-test-slave-fmv8 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:31:36 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:36 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:31:36.813496 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:36.806489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.0.98.62:45023)
0929 18:31:36.806633 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.806638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.806852 (+ 214us) server_negotiation.cc:407] Connection header received
0929 18:31:36.806961 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.806964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.807013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.807108 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:36.807828 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.808669 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.809553 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.809927 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.810052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.813349 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35014: BlockingRecv error: recv got EOF from 127.0.0.1:35014 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:31:36 dist-test-slave-fmv8 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 18:31:36.823621 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:36.815132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.98.62:45023)
0929 18:31:36.815255 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:36.815261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:36.815524 (+ 263us) server_negotiation.cc:407] Connection header received
0929 18:31:36.815695 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:36.815698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:36.815738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:36.815822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:36.816484 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.816973 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.817754 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:36.817979 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:36.818087 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:36.820582 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:36.820586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:36.820589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:36.820615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:36.822033 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:36.822441 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:36.822446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:36.822447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:36.822533 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:36.822806 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:36.822810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:36.822812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:36.822983 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:36.823114 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:36.823348 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 18:31:36.823455 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:31:36.826342 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:31:36.826776 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:36.831630 8033 tablet_service.cc:1467] Processing CreateTablet for tablet a657abb21d6f42ed888f20172d9a605c (DEFAULT_TABLE table=test-table [id=82b87babe2d54d94ba298614fd74e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.831804 8313 tablet_service.cc:1467] Processing CreateTablet for tablet a657abb21d6f42ed888f20172d9a605c (DEFAULT_TABLE table=test-table [id=82b87babe2d54d94ba298614fd74e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.831965 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a657abb21d6f42ed888f20172d9a605c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.832105 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a657abb21d6f42ed888f20172d9a605c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.833935 8412 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Bootstrap starting.
I20260929 18:31:36.834177 8173 tablet_service.cc:1467] Processing CreateTablet for tablet a657abb21d6f42ed888f20172d9a605c (DEFAULT_TABLE table=test-table [id=82b87babe2d54d94ba298614fd74e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:36.834446 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a657abb21d6f42ed888f20172d9a605c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:36.834631 8408 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Bootstrap starting.
I20260929 18:31:36.834859 8412 tablet_bootstrap.cc:654] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.835815 8408 tablet_bootstrap.cc:654] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.836772 8407 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Bootstrap starting.
I20260929 18:31:36.837013 8408 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: No bootstrap required, opened a new log
I20260929 18:31:36.837082 8408 ts_tablet_manager.cc:1397] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:31:36.837476 8408 raft_consensus.cc:348] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.837565 8407 tablet_bootstrap.cc:654] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:36.837579 8408 raft_consensus.cc:374] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.837608 8408 raft_consensus.cc:729] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926b08b0e0994468afcd33c8220501df, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.837733 8408 consensus_queue.cc:260] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.837982 8408 ts_tablet_manager.cc:1428] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:36.838913 8412 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: No bootstrap required, opened a new log
I20260929 18:31:36.838999 8412 ts_tablet_manager.cc:1397] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:31:36.839046 8407 tablet_bootstrap.cc:492] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: No bootstrap required, opened a new log
I20260929 18:31:36.839118 8407 ts_tablet_manager.cc:1397] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:36.839463 8407 raft_consensus.cc:348] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.839570 8407 raft_consensus.cc:374] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.839465 8412 raft_consensus.cc:348] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.839602 8407 raft_consensus.cc:729] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.839639 8412 raft_consensus.cc:374] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:36.839694 8412 raft_consensus.cc:729] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Initialized, Role: FOLLOWER
I20260929 18:31:36.839726 8407 consensus_queue.cc:260] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.839816 8412 consensus_queue.cc:260] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:36.839955 8407 ts_tablet_manager.cc:1428] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:36.840121 8412 ts_tablet_manager.cc:1428] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:37.035578 8467 raft_consensus.cc:482] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:37.035710 8467 raft_consensus.cc:504] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.036279 8467 leader_election.cc:288] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:37.036724 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a657abb21d6f42ed888f20172d9a605c" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df" is_pre_election: true
I20260929 18:31:37.036722 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a657abb21d6f42ed888f20172d9a605c" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9" is_pre_election: true
I20260929 18:31:37.036881 8185 raft_consensus.cc:2454] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:37.036881 8332 raft_consensus.cc:2454] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:37.037195 7985 leader_election.cc:302] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:37.037385 8467 raft_consensus.cc:2781] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:37.037446 8467 raft_consensus.cc:482] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:37.037473 8467 raft_consensus.cc:3037] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.038427 8467 raft_consensus.cc:504] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.038755 8467 leader_election.cc:288] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:37.039049 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a657abb21d6f42ed888f20172d9a605c" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9"
I20260929 18:31:37.039178 8185 raft_consensus.cc:3037] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.039152 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a657abb21d6f42ed888f20172d9a605c" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df"
I20260929 18:31:37.039350 8332 raft_consensus.cc:3037] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.040256 8332 raft_consensus.cc:2454] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:37.040346 8185 raft_consensus.cc:2454] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:37.040544 7985 leader_election.cc:302] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:37.040719 8467 raft_consensus.cc:2781] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:37.040835 8467 raft_consensus.cc:686] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Running, Role: LEADER
I20260929 18:31:37.040977 8467 consensus_queue.cc:237] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.042672 7894 catalog_manager.cc:5495] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1088a52a955e42e78f35e53c52090ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:37.079838 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:37.076815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.0.98.1:34157)
0929 18:31:37.076933 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:37.076937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:37.076949 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:31:37.077059 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:37.077061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:37.077102 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:37.077204 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:31:37.077447 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:37.077940 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:37.078899 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:37.079107 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:37.079226 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:37.079488 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:31:37.079550 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:31:37.079603 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:37.079698 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:31:37.079735 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:31:37.081645 8185 raft_consensus.cc:1261] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Refusing update from remote peer 1088a52a955e42e78f35e53c52090ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:37.081672 8332 raft_consensus.cc:1261] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Refusing update from remote peer 1088a52a955e42e78f35e53c52090ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:37.082054 8467 consensus_queue.cc:1035] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:37.082218 8438 consensus_queue.cc:1035] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:37.089219 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:37.089359 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:37.091559 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Sending DeleteTablet for 3 replicas of tablet a657abb21d6f42ed888f20172d9a605c
I20260929 18:31:37.092063 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet a657abb21d6f42ed888f20172d9a605c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60282
I20260929 18:31:37.092206 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet a657abb21d6f42ed888f20172d9a605c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260929 18:31:37.092324 8472 tablet_replica.cc:330] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: stopping tablet replica
I20260929 18:31:37.092363 8471 tablet_replica.cc:330] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: stopping tablet replica
I20260929 18:31:37.092434 8472 raft_consensus.cc:2229] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:37.092430 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet a657abb21d6f42ed888f20172d9a605c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54626
I20260929 18:31:37.092471 8471 raft_consensus.cc:2229] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:37.092594 8473 tablet_replica.cc:330] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: stopping tablet replica
I20260929 18:31:37.092592 8472 raft_consensus.cc:2258] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.092590 8471 raft_consensus.cc:2258] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.092697 8473 raft_consensus.cc:2229] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:37.092816 8473 raft_consensus.cc:2258] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.093243 8471 ts_tablet_manager.cc:1905] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:37.093258 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:31:37.093499 8473 ts_tablet_manager.cc:1905] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:31:37.093641 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:31:37.095515 8471 ts_tablet_manager.cc:1918] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:37.095613 8471 log.cc:1196] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/a657abb21d6f42ed888f20172d9a605c
I20260929 18:31:37.095854 8471 ts_tablet_manager.cc:1939] T a657abb21d6f42ed888f20172d9a605c P 926b08b0e0994468afcd33c8220501df: Deleting consensus metadata
I20260929 18:31:37.095964 8473 ts_tablet_manager.cc:1918] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:37.096042 8473 log.cc:1196] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/a657abb21d6f42ed888f20172d9a605c
I20260929 18:31:37.096335 8473 ts_tablet_manager.cc:1939] T a657abb21d6f42ed888f20172d9a605c P 343a15104d92409a8f5249e4268e45b9: Deleting consensus metadata
I20260929 18:31:37.096589 8472 ts_tablet_manager.cc:1905] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:37.096940 7882 catalog_manager.cc:4841] TS 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865): tablet a657abb21d6f42ed888f20172d9a605c (table test-table [id=82b87babe2d54d94ba298614fd74e02e]) successfully deleted
I20260929 18:31:37.098099 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf (DEFAULT_TABLE table=test-table [id=7cfd354973794c2cb015d87b7406bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:37.098320 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9493ba803e404b9fc2f2f20a54abdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:37.098263 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf (DEFAULT_TABLE table=test-table [id=7cfd354973794c2cb015d87b7406bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:37.098462 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9493ba803e404b9fc2f2f20a54abdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:37.098786 8472 ts_tablet_manager.cc:1918] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:31:37.098846 8472 log.cc:1196] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/a657abb21d6f42ed888f20172d9a605c
I20260929 18:31:37.099076 8472 ts_tablet_manager.cc:1939] T a657abb21d6f42ed888f20172d9a605c P 1088a52a955e42e78f35e53c52090ab3: Deleting consensus metadata
I20260929 18:31:37.099658 7882 catalog_manager.cc:4841] TS 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157): tablet a657abb21d6f42ed888f20172d9a605c (table test-table [id=82b87babe2d54d94ba298614fd74e02e]) successfully deleted
I20260929 18:31:37.099795 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf (DEFAULT_TABLE table=test-table [id=7cfd354973794c2cb015d87b7406bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:31:37.099998 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9493ba803e404b9fc2f2f20a54abdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:37.100205 8408 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Bootstrap starting.
I20260929 18:31:37.100659 8407 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Bootstrap starting.
I20260929 18:31:37.101135 8408 tablet_bootstrap.cc:654] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:37.101464 8407 tablet_bootstrap.cc:654] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:37.102131 8408 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: No bootstrap required, opened a new log
I20260929 18:31:37.102161 8412 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: Bootstrap starting.
I20260929 18:31:37.102236 8408 ts_tablet_manager.cc:1397] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:31:37.102401 8407 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: No bootstrap required, opened a new log
I20260929 18:31:37.102484 8407 ts_tablet_manager.cc:1397] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:37.102674 8408 raft_consensus.cc:348] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.102784 8408 raft_consensus.cc:374] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:37.102823 8408 raft_consensus.cc:729] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926b08b0e0994468afcd33c8220501df, State: Initialized, Role: FOLLOWER
I20260929 18:31:37.102836 8407 raft_consensus.cc:348] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.102919 8412 tablet_bootstrap.cc:654] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:37.102963 8407 raft_consensus.cc:374] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:37.103003 8407 raft_consensus.cc:729] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a15104d92409a8f5249e4268e45b9, State: Initialized, Role: FOLLOWER
I20260929 18:31:37.102958 8408 consensus_queue.cc:260] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.103132 8407 consensus_queue.cc:260] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.103240 8408 ts_tablet_manager.cc:1428] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:31:37.103410 8407 ts_tablet_manager.cc:1428] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:31:37.104457 8412 tablet_bootstrap.cc:492] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: No bootstrap required, opened a new log
I20260929 18:31:37.104573 8412 ts_tablet_manager.cc:1397] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:37.104977 8412 raft_consensus.cc:348] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.105084 8412 raft_consensus.cc:374] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:37.105116 8412 raft_consensus.cc:729] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Initialized, Role: FOLLOWER
I20260929 18:31:37.105235 8412 consensus_queue.cc:260] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.105506 8412 ts_tablet_manager.cc:1428] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:31:37.106141 7881 catalog_manager.cc:4841] TS 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035): tablet a657abb21d6f42ed888f20172d9a605c (table test-table [id=82b87babe2d54d94ba298614fd74e02e]) successfully deleted
I20260929 18:31:37.245386 8467 raft_consensus.cc:482] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:31:37.245520 8467 raft_consensus.cc:504] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.245874 8467 leader_election.cc:288] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:37.246330 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9493ba803e404b9fc2f2f20a54abdf" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df" is_pre_election: true
I20260929 18:31:37.246330 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9493ba803e404b9fc2f2f20a54abdf" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9" is_pre_election: true
I20260929 18:31:37.246496 8332 raft_consensus.cc:2454] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:37.246510 8185 raft_consensus.cc:2454] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 0.
I20260929 18:31:37.246868 7985 leader_election.cc:302] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:37.247097 8467 raft_consensus.cc:2781] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:31:37.247159 8467 raft_consensus.cc:482] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:31:37.247201 8467 raft_consensus.cc:3037] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.248126 8467 raft_consensus.cc:504] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.248487 8467 leader_election.cc:288] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865), 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035)
I20260929 18:31:37.248805 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9493ba803e404b9fc2f2f20a54abdf" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a15104d92409a8f5249e4268e45b9"
I20260929 18:31:37.248828 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9493ba803e404b9fc2f2f20a54abdf" candidate_uuid: "1088a52a955e42e78f35e53c52090ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926b08b0e0994468afcd33c8220501df"
I20260929 18:31:37.248939 8185 raft_consensus.cc:3037] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.248953 8332 raft_consensus.cc:3037] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:37.249794 8332 raft_consensus.cc:2454] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:37.249902 8185 raft_consensus.cc:2454] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1088a52a955e42e78f35e53c52090ab3 in term 1.
I20260929 18:31:37.250131 7985 leader_election.cc:302] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1088a52a955e42e78f35e53c52090ab3, 926b08b0e0994468afcd33c8220501df; no voters:
I20260929 18:31:37.250278 8467 raft_consensus.cc:2781] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:37.250368 8467 raft_consensus.cc:686] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 1088a52a955e42e78f35e53c52090ab3, State: Running, Role: LEADER
I20260929 18:31:37.250507 8467 consensus_queue.cc:237] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } }
I20260929 18:31:37.252218 7896 catalog_manager.cc:5495] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1088a52a955e42e78f35e53c52090ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1088a52a955e42e78f35e53c52090ab3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 } health_report { overall_health: UNKNOWN } } }
I20260929 18:31:37.270300 8332 raft_consensus.cc:1261] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Refusing update from remote peer 1088a52a955e42e78f35e53c52090ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:37.270331 8185 raft_consensus.cc:1261] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Refusing update from remote peer 1088a52a955e42e78f35e53c52090ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:31:37.270771 8415 consensus_queue.cc:1035] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a15104d92409a8f5249e4268e45b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:37.270916 8467 consensus_queue.cc:1035] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926b08b0e0994468afcd33c8220501df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:31:37.274111 8422 mvcc.cc:205] Tried to move back new op lower bound from 7334734632015740928 to 7334734631939473408. Current Snapshot: MvccSnapshot[applied={T|T < 7334734632015740928}]
I20260929 18:31:37.299491 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:37.299654 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:31:37.301527 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41c4ba959524f1fa81371acd612b072: Sending DeleteTablet for 3 replicas of tablet 8d9493ba803e404b9fc2f2f20a54abdf
I20260929 18:31:37.302007 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60282
I20260929 18:31:37.302064 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54626
I20260929 18:31:37.302191 8472 tablet_replica.cc:330] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3: stopping tablet replica
I20260929 18:31:37.302254 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9493ba803e404b9fc2f2f20a54abdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:31:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260929 18:31:37.302341 8472 raft_consensus.cc:2229] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:31:37.302429 8471 tablet_replica.cc:330] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: stopping tablet replica
I20260929 18:31:37.302430 8473 tablet_replica.cc:330] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: stopping tablet replica
I20260929 18:31:37.302525 8472 raft_consensus.cc:2258] T 8d9493ba803e404b9fc2f2f20a54abdf P 1088a52a955e42e78f35e53c52090ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.302529 8471 raft_consensus.cc:2229] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:37.302557 8473 raft_consensus.cc:2229] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:31:37.302632 8471 raft_consensus.cc:2258] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.302659 8473 raft_consensus.cc:2258] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:31:37.302814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7968
I20260929 18:31:37.304073 8473 ts_tablet_manager.cc:1905] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:37.304314 8471 ts_tablet_manager.cc:1905] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:31:37.306236 8473 ts_tablet_manager.cc:1918] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:37.306332 8473 log.cc:1196] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/8d9493ba803e404b9fc2f2f20a54abdf
I20260929 18:31:37.306564 8473 ts_tablet_manager.cc:1939] T 8d9493ba803e404b9fc2f2f20a54abdf P 343a15104d92409a8f5249e4268e45b9: Deleting consensus metadata
I20260929 18:31:37.307349 8471 ts_tablet_manager.cc:1918] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:31:37.307435 8471 log.cc:1196] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/8d9493ba803e404b9fc2f2f20a54abdf
I20260929 18:31:37.307377 7882 catalog_manager.cc:4841] TS 343a15104d92409a8f5249e4268e45b9 (127.0.98.2:37865): tablet 8d9493ba803e404b9fc2f2f20a54abdf (table test-table [id=7cfd354973794c2cb015d87b7406bf25]) successfully deleted
I20260929 18:31:37.307687 8471 ts_tablet_manager.cc:1939] T 8d9493ba803e404b9fc2f2f20a54abdf P 926b08b0e0994468afcd33c8220501df: Deleting consensus metadata
I20260929 18:31:37.308394 7881 catalog_manager.cc:4841] TS 926b08b0e0994468afcd33c8220501df (127.0.98.3:36035): tablet 8d9493ba803e404b9fc2f2f20a54abdf (table test-table [id=7cfd354973794c2cb015d87b7406bf25]) successfully deleted
W20260929 18:31:37.311429 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34157 (error 108)
I20260929 18:31:37.312191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8108
I20260929 18:31:37.313145 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:37.312076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34157 (local address 127.0.0.1:60308)
0929 18:31:37.312333 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:37.313021 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
W20260929 18:31:37.313265 7882 catalog_manager.cc:4584] TS 1088a52a955e42e78f35e53c52090ab3 (127.0.98.1:34157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d9493ba803e404b9fc2f2f20a54abdf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34157: connect: Connection refused (error 111)
I20260929 18:31:37.319239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8248
I20260929 18:31:37.325014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 7864
2026-09-29T18:31:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3477 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:37 dist-test-slave-fmv8 krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:37 dist-test-slave-fmv8 krb5kdc[8506](info): set up 2 sockets
Sep 29 18:31:37 dist-test-slave-fmv8 krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260929 18:31:38.356905 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:38 dist-test-slave-fmv8 krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:38.475081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:38.580920 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:38.581218 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:38.581300 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:38.585219 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:38.585309 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:38.585347 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:38.585383 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:38.585418 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:38.590785 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:38.591851 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:38.592856 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:38.601188 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:38.601408 8519 server_base.cc:1034] running on GCE node
W20260929 18:31:38.601186 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:38.601166 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:38.607229 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:38.608744 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:38.609926 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790706698609885 us; error 58 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:31:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1296 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:38 dist-test-slave-fmv8 krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:38 dist-test-slave-fmv8 krb5kdc[8532](info): set up 2 sockets
Sep 29 18:31:38 dist-test-slave-fmv8 krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260929 18:31:39.656945 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:39 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:39.773074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:39.880978 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:39.881296 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:39.881397 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:39.885439 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:39.885531 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:39.885571 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:39.885629 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:39.885668 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:39.891714 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:39.892846 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:39.893886 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:39.902269 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:39.902280 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:39.902261 8545 server_base.cc:1034] running on GCE node
W20260929 18:31:39.902241 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:39.903072 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:39.904069 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:39.905264 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790706699905245 us; error 31 us; skew 500 ppm
Sep 29 18:31:39 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:39.912425 8545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:39.913288 8545 webserver.cc:460] Webserver started at http://127.0.98.62:42345/ using document root <none> and password file <none>
I20260929 18:31:39.913837 8545 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:39.913926 8545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:39.914153 8545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:39.916184 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "dc27368e5d464650a2a763f4a2e7907b"
format_stamp: "Formatted at 2026-09-29 18:31:39 on dist-test-slave-fmv8"
server_key: "2bf54fb9b1e39b18b1dfd46bb4bd60d1"
server_key_iv: "c00a557299720983072f73010bfaa4a1"
server_key_version: "encryptionkey@0"
I20260929 18:31:39.916755 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc27368e5d464650a2a763f4a2e7907b"
format_stamp: "Formatted at 2026-09-29 18:31:39 on dist-test-slave-fmv8"
server_key: "2bf54fb9b1e39b18b1dfd46bb4bd60d1"
server_key_iv: "c00a557299720983072f73010bfaa4a1"
server_key_version: "encryptionkey@0"
I20260929 18:31:39.920627 8545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:31:39.923070 8560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:39.924338 8545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:39.924490 8545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "dc27368e5d464650a2a763f4a2e7907b"
format_stamp: "Formatted at 2026-09-29 18:31:39 on dist-test-slave-fmv8"
server_key: "2bf54fb9b1e39b18b1dfd46bb4bd60d1"
server_key_iv: "c00a557299720983072f73010bfaa4a1"
server_key_version: "encryptionkey@0"
I20260929 18:31:39.924602 8545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:39.953377 8545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:39.956350 8545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:39.956554 8545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:39.963660 8545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41865
I20260929 18:31:39.963685 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41865 every 8 connection(s)
I20260929 18:31:39.964659 8545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:39.967207 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:39.969012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8545
I20260929 18:31:39.969106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:39.969412 392 external_mini_cluster.cc:1428] Setting key 01df65939bc9b1329bf5fe419e974afb
I20260929 18:31:39.973883 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Bootstrap starting.
I20260929 18:31:39.976385 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:39.977142 8613 log.cc:824] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:31:39 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:39.979281 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: No bootstrap required, opened a new log
I20260929 18:31:39.983880 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:39.984125 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:39.984194 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc27368e5d464650a2a763f4a2e7907b, State: Initialized, Role: FOLLOWER
I20260929 18:31:39.984464 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:39.970820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.0.98.62:41865)
0929 18:31:39.971233 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:39.971251 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:39.971292 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:31:39.972050 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:39.972073 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:39.972494 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:39.972847 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:39.973524 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:39.974909 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:39.975757 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:39.976490 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:39.976664 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:39.979418 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:39.979424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:39.979439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:39.979467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:39.981442 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:39.981854 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:39.981860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:39.981866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:39.982009 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:39.982312 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:39.982316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:39.982318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:39.982789 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:39.982983 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:39.983228 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:31:39.983515 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260929 18:31:39.984666 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:39.984819 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:39.984882 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:39.984963 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:39.985934 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:39.986268 8613 leader_election.cc:302] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc27368e5d464650a2a763f4a2e7907b; no voters:
I20260929 18:31:39.986577 8613 leader_election.cc:288] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:39.986871 8618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:39.987098 8618 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 LEADER]: Becoming Leader. State: Replica: dc27368e5d464650a2a763f4a2e7907b, State: Running, Role: LEADER
I20260929 18:31:39.987392 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:39.987780 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:39.988940 8619 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc27368e5d464650a2a763f4a2e7907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } } }
I20260929 18:31:39.989066 8619 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:39.989486 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:39.989449 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc27368e5d464650a2a763f4a2e7907b. Latest consensus state: current_term: 1 leader_uuid: "dc27368e5d464650a2a763f4a2e7907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } } }
I20260929 18:31:39.989611 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:39.992412 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:39.998344 8627 catalog_manager.cc:1347] Generated new cluster ID: 54d92642c42d47e2931d6d14ddd77ef3
I20260929 18:31:39.998436 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:40.015170 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:40.016479 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:40.029544 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Generated new TSK 0
I20260929 18:31:40.030382 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:40.065462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:40.171644 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:40.171881 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:40.171923 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:40.175858 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:40.175915 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:40.176007 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:40.181035 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:40.182036 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:40.183024 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:40.192752 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.192752 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.192747 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:40.193071 8639 server_base.cc:1034] running on GCE node
I20260929 18:31:40.193470 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:40.194115 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:40.195297 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790706700195262 us; error 47 us; skew 500 ppm
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:40.198016 8639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:40.198835 8639 webserver.cc:460] Webserver started at http://127.0.98.1:43393/ using document root <none> and password file <none>
I20260929 18:31:40.199398 8639 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:40.199491 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:40.199718 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:40.201797 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "e72d9810a5ed4760b671e32418b4da31"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6c5771ba6363bf7fc51b3764c595aa7b"
server_key_iv: "aa1715ecd4892370c2f0fd9bd67c9bb4"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.202355 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e72d9810a5ed4760b671e32418b4da31"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6c5771ba6363bf7fc51b3764c595aa7b"
server_key_iv: "aa1715ecd4892370c2f0fd9bd67c9bb4"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.205885 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:40.208309 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.209527 8639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:31:40.209623 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "e72d9810a5ed4760b671e32418b4da31"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6c5771ba6363bf7fc51b3764c595aa7b"
server_key_iv: "aa1715ecd4892370c2f0fd9bd67c9bb4"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.209702 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:40.230280 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:40.233037 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:40.233179 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:40.233662 8639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:40.234373 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:40.234416 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.234455 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:40.234470 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.243227 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39113
I20260929 18:31:40.243261 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39113 every 8 connection(s)
I20260929 18:31:40.244057 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:40.251461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8639
I20260929 18:31:40.251585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:40.251904 392 external_mini_cluster.cc:1428] Setting key 467d5b9049499555ef311d4eefbf8051
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:40.258183 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.245821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47999 (local address 127.0.98.62:41865)
0929 18:31:40.245980 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:40.245985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:40.246753 (+ 768us) server_negotiation.cc:407] Connection header received
0929 18:31:40.248059 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:40.248062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:40.248110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:40.248245 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:40.249464 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.250063 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.251225 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.251590 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.251700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.254266 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:40.254273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:40.254279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:40.254311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:40.255838 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.256491 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.256494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.256495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.256568 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.256951 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.256953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.256954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.257109 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:40.257183 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:40.257848 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 18:31:40.257980 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:31:40.259279 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.246153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:47999)
0929 18:31:40.246607 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:40.246650 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:40.247878 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:40.248497 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:40.248504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:40.248844 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:40.249313 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.249327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.250200 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.250202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.251087 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.251096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.251773 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.251776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.251932 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.252615 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:40.252637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:40.254114 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:40.255950 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.255959 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.255975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.256400 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.256660 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.256666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.256671 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.256849 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.257219 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:40.257227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:40.257565 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 18:31:40.257782 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 18:31:40.258145 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260929 18:31:40.260563 8768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41865
I20260929 18:31:40.260792 8768 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:40.261242 8768 heartbeater.cc:511] Master 127.0.98.62:41865 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:40.262671 8577 ts_manager.cc:194] Registered new tserver with Master: e72d9810a5ed4760b671e32418b4da31 (127.0.98.1:39113)
I20260929 18:31:40.263845 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:40.278468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:40.384205 8773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:40.384479 8773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:40.384555 8773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:40.388863 8773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:40.388932 8773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:40.389050 8773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:40.394197 8773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:40.395198 8773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:40.396246 8773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:40.405625 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.405637 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.405634 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:40.406001 8773 server_base.cc:1034] running on GCE node
I20260929 18:31:40.406411 8773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:40.407090 8773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:40.408244 8773 hybrid_clock.cc:648] HybridClock initialized: now 1790706700408208 us; error 38 us; skew 500 ppm
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:40.410936 8773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:40.411736 8773 webserver.cc:460] Webserver started at http://127.0.98.2:33253/ using document root <none> and password file <none>
I20260929 18:31:40.412425 8773 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:40.412525 8773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:40.412729 8773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:40.414826 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "27da36d459814ac28e0efce893e79130"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "32b657c7555a3bf56505ef0952e2681f"
server_key_iv: "864168d922c592772a8ff2a9a52daf60"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.415390 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27da36d459814ac28e0efce893e79130"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "32b657c7555a3bf56505ef0952e2681f"
server_key_iv: "864168d922c592772a8ff2a9a52daf60"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.419013 8773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:31:40.421604 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.422931 8773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:40.423048 8773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "27da36d459814ac28e0efce893e79130"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "32b657c7555a3bf56505ef0952e2681f"
server_key_iv: "864168d922c592772a8ff2a9a52daf60"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.423172 8773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:40.453509 8773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:40.456374 8773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:40.456578 8773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:40.457160 8773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:40.457953 8773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:40.458022 8773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.458091 8773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:40.458144 8773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.466598 8773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45789
I20260929 18:31:40.466668 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45789 every 8 connection(s)
I20260929 18:31:40.467461 8773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:40.474519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8773
I20260929 18:31:40.474660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:40.474980 392 external_mini_cluster.cc:1428] Setting key 189c7ded7f7011df4f2fc52378c84235
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:40.480873 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.469170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39987 (local address 127.0.98.62:41865)
0929 18:31:40.469304 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:40.469308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:40.470107 (+ 799us) server_negotiation.cc:407] Connection header received
0929 18:31:40.471367 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:40.471369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:40.471418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:40.471492 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:40.472640 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.473154 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.474194 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.474414 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.474489 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.477147 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:40.477154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:40.477160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:40.477193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:40.478713 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.479309 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.479312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.479314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.479406 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.479791 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.479793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.479794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.479958 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:40.480033 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:40.480665 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 18:31:40.480763 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:31:40.482048 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.469532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:39987)
0929 18:31:40.469963 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:40.470003 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:40.471190 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:40.471647 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:40.471654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:40.471999 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:40.472519 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.472531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.473253 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.473256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.474091 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.474099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.474565 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.474571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.474776 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.475336 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:40.475359 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:40.476988 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:40.478830 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.478835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.478848 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.479205 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.479503 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.479509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.479514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.479699 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.480063 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:40.480072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:40.480410 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 18:31:40.480632 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 18:31:40.480972 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260929 18:31:40.483206 8902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41865
I20260929 18:31:40.483443 8902 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:40.483886 8902 heartbeater.cc:511] Master 127.0.98.62:41865 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:31:40.484921 8577 ts_manager.cc:194] Registered new tserver with Master: 27da36d459814ac28e0efce893e79130 (127.0.98.2:45789)
I20260929 18:31:40.485471 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:31:40.501415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:40.612828 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:40.613052 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:40.613123 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:40.617238 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:40.617288 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:40.617381 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:40.622421 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:40.623479 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:40.624518 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:40.633862 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.633930 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.633884 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:40.634162 8907 server_base.cc:1034] running on GCE node
I20260929 18:31:40.634521 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:40.635128 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:40.636305 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790706700636278 us; error 48 us; skew 500 ppm
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:40.638962 8907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:31:40.639746 8907 webserver.cc:460] Webserver started at http://127.0.98.3:35165/ using document root <none> and password file <none>
I20260929 18:31:40.640360 8907 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:40.640447 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:40.640666 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:40.642592 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ceec59e6dd24b0c8ecc0f4b7ccf3c1e"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6a7225ed13217a3eb49028ab3ed35b20"
server_key_iv: "268ef8cfce68809995e1b902c282d667"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.643180 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ceec59e6dd24b0c8ecc0f4b7ccf3c1e"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6a7225ed13217a3eb49028ab3ed35b20"
server_key_iv: "268ef8cfce68809995e1b902c282d667"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.646749 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:40.649073 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.650238 8907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:40.650365 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "6ceec59e6dd24b0c8ecc0f4b7ccf3c1e"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6a7225ed13217a3eb49028ab3ed35b20"
server_key_iv: "268ef8cfce68809995e1b902c282d667"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.650468 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:40.668044 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:40.670787 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:40.670990 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:40.671535 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:40.672360 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:40.672428 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.672495 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:40.672539 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.681324 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34603
I20260929 18:31:40.681348 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34603 every 8 connection(s)
I20260929 18:31:40.682157 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:40.688191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 8907
I20260929 18:31:40.688349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:40.688656 392 external_mini_cluster.cc:1428] Setting key 40580fc7390b50149eba028114f9710a
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:40.695284 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.683956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48477 (local address 127.0.98.62:41865)
0929 18:31:40.684089 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:40.684093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:40.684888 (+ 795us) server_negotiation.cc:407] Connection header received
0929 18:31:40.686181 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:40.686184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:40.686228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:40.686303 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:40.687450 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.688049 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.689588 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.689872 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.690009 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.692338 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:40.692341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:40.692343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:40.692364 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:40.693570 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.694035 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.694038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.694039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.694107 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:40.694385 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:40.694387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:40.694388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:40.694542 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:40.694660 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:40.695090 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 18:31:40.695178 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:31:40.695999 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:40.684272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:48477)
0929 18:31:40.684735 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:40.684777 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:40.685998 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:40.686471 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:40.686478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:40.686865 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:40.687318 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.687331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.688241 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.688247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.689470 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:40.689478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:40.690044 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:40.690048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:40.690208 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:40.690800 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:40.690818 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:40.692192 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:40.693666 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.693670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.693680 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.693944 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.694178 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:40.694180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:40.694181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:40.694303 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:40.694653 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:40.694658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:40.694878 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 18:31:40.695021 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:31:40.695237 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260929 18:31:40.697037 9036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41865
I20260929 18:31:40.697250 9036 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:40.697680 9036 heartbeater.cc:511] Master 127.0.98.62:41865 requested a full tablet report, sending...
I20260929 18:31:40.698596 8577 ts_manager.cc:194] Registered new tserver with Master: 6ceec59e6dd24b0c8ecc0f4b7ccf3c1e (127.0.98.3:34603)
I20260929 18:31:40.699086 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48477
I20260929 18:31:40.702215 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 18:31:40.769004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8639
I20260929 18:31:40.774510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8773
I20260929 18:31:40.779842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8907
I20260929 18:31:40.784919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 8545
I20260929 18:31:40.791028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41865
--webserver_interface=127.0.98.62
--webserver_port=42345
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:40.901868 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:40.902129 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:40.902168 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:40.906185 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:40.906247 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:40.906272 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:40.906289 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:40.906328 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:40.911643 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:40.912711 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:40.913723 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:40.922020 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.921913 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:40.921917 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:40.922156 9045 server_base.cc:1034] running on GCE node
I20260929 18:31:40.922798 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:40.923981 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:40.925199 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790706700925189 us; error 62 us; skew 500 ppm
Sep 29 18:31:40 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:40.927968 9045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:40.928874 9045 webserver.cc:460] Webserver started at http://127.0.98.62:42345/ using document root <none> and password file <none>
I20260929 18:31:40.929440 9045 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:40.929548 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:40.933331 9045 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260929 18:31:40.935385 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:40.936719 9045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:40.936869 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "dc27368e5d464650a2a763f4a2e7907b"
format_stamp: "Formatted at 2026-09-29 18:31:39 on dist-test-slave-fmv8"
server_key: "2bf54fb9b1e39b18b1dfd46bb4bd60d1"
server_key_iv: "c00a557299720983072f73010bfaa4a1"
server_key_version: "encryptionkey@0"
I20260929 18:31:40.937427 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:40.961225 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:40.964489 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:40.964694 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:40.971807 9045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41865
I20260929 18:31:40.971843 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41865 every 8 connection(s)
I20260929 18:31:40.972860 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:40.976956 9113 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:31:40.977118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9045
I20260929 18:31:40.977553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39113
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41865
--webserver_port=43393
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:31:40.978813 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Bootstrap starting.
I20260929 18:31:41.001469 9113 log.cc:824] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:41.009505 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:31:41.009922 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Bootstrap complete.
I20260929 18:31:41.015334 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:41.015810 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc27368e5d464650a2a763f4a2e7907b, State: Initialized, Role: FOLLOWER
I20260929 18:31:41.016377 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:41.016518 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:41.016582 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:41.016669 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:31:41.018966 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:41.019327 9113 leader_election.cc:302] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc27368e5d464650a2a763f4a2e7907b; no voters:
I20260929 18:31:41.019693 9113 leader_election.cc:288] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:31:41.019830 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:31:41.020107 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [term 2 LEADER]: Becoming Leader. State: Replica: dc27368e5d464650a2a763f4a2e7907b, State: Running, Role: LEADER
I20260929 18:31:41.020448 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } }
I20260929 18:31:41.020936 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:41.021301 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc27368e5d464650a2a763f4a2e7907b. Latest consensus state: current_term: 2 leader_uuid: "dc27368e5d464650a2a763f4a2e7907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } } }
I20260929 18:31:41.021428 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:41.021726 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc27368e5d464650a2a763f4a2e7907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc27368e5d464650a2a763f4a2e7907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41865 } } }
I20260929 18:31:41.021816 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:41.021991 9121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:41.025810 9121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:41.026423 9121 catalog_manager.cc:1259] Loaded cluster ID: 54d92642c42d47e2931d6d14ddd77ef3
I20260929 18:31:41.026484 9121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:41.028774 9121 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:41.029429 9121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc27368e5d464650a2a763f4a2e7907b: Loaded TSK: 0
I20260929 18:31:41.030635 9121 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:31:41.101078 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:41.101377 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:41.101452 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:41.105501 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:41.105587 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:41.105674 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:41.110843 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43393
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:41.111838 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:41.112823 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:41.123066 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:41.123067 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:41.123304 9115 server_base.cc:1034] running on GCE node
W20260929 18:31:41.123149 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:41.123767 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:41.124528 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:41.125706 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790706701125690 us; error 33 us; skew 500 ppm
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706701, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:41.128437 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 18:31:41.129340 9115 webserver.cc:460] Webserver started at http://127.0.98.1:43393/ using document root <none> and password file <none>
I20260929 18:31:41.129879 9115 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:41.129951 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:41.133739 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:41.135704 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.136946 9115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:41.137035 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "e72d9810a5ed4760b671e32418b4da31"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6c5771ba6363bf7fc51b3764c595aa7b"
server_key_iv: "aa1715ecd4892370c2f0fd9bd67c9bb4"
server_key_version: "encryptionkey@0"
I20260929 18:31:41.137569 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:41.158077 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:41.161350 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:41.161556 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:41.162186 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:41.162984 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:41.163031 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.163100 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:41.163129 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.172248 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39113
I20260929 18:31:41.172273 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39113 every 8 connection(s)
I20260929 18:31:41.173177 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:41.174363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9115
I20260929 18:31:41.174881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45789
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41865
--webserver_port=33253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.189698 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.175545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:35619)
0929 18:31:41.176237 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.176309 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.178407 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.179576 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.179588 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.180191 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.180955 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.180980 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.182254 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.182261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.184000 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.184016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.184887 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.184894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.185144 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.185797 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.185826 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.188738 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":126,"threads_started":1}
W20260929 18:31:41.189697 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.175516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35619 (local address 127.0.98.62:41865)
0929 18:31:41.175987 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.176008 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.176455 (+ 447us) server_negotiation.cc:407] Connection header received
0929 18:31:41.178837 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.178849 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.179136 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.179424 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.181144 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.182106 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.184213 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.184701 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.184836 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.188951 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35619: BlockingRecv error: recv got EOF from 127.0.98.1:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
W20260929 18:31:41.190172 9263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41865 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41865: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.195892 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.190672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:44489)
0929 18:31:41.190943 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.190959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.191074 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.191342 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.191344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.191509 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.191596 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.191602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.192380 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.192386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.192931 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.192938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.193376 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.193379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.193467 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.193954 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.193965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.195759 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200}
W20260929 18:31:41.195981 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.190798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44489 (local address 127.0.98.62:41865)
0929 18:31:41.190916 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.190920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.191029 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:31:41.191180 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.191183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.191234 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.191328 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.191709 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.192281 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.193049 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.193270 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.193375 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.195869 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44489: BlockingRecv error: recv got EOF from 127.0.98.1:44489 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.201084 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.196450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:56241)
0929 18:31:41.196555 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.196571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.196641 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.196892 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.196895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.197044 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.197104 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.197108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.197801 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.197805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.198312 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.198319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.198736 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.198740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.198818 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.199269 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.199280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.200951 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:31:41.201167 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.196522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56241 (local address 127.0.98.62:41865)
0929 18:31:41.196637 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.196641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.196654 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:41.196718 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.196722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.196768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.196879 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.197208 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.197683 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.198426 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.198633 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.198736 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.201056 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56241: BlockingRecv error: recv got EOF from 127.0.98.1:56241 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:31:41.201457 9263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:31:41.291965 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:41.292236 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:41.292279 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:41.296118 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:41.296180 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:41.296241 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:41.301295 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33253
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:41.302219 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:41.303124 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:41.312515 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:41.312539 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:41.312539 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:41.312616 9267 server_base.cc:1034] running on GCE node
I20260929 18:31:41.313273 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:41.313928 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:41.315133 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790706701315112 us; error 34 us; skew 500 ppm
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706701, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:41.318118 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 18:31:41.319099 9267 webserver.cc:460] Webserver started at http://127.0.98.2:33253/ using document root <none> and password file <none>
I20260929 18:31:41.319674 9267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:41.319785 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:41.324362 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:41.326488 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.327646 9267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:41.327787 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "27da36d459814ac28e0efce893e79130"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "32b657c7555a3bf56505ef0952e2681f"
server_key_iv: "864168d922c592772a8ff2a9a52daf60"
server_key_version: "encryptionkey@0"
I20260929 18:31:41.328392 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:41.355010 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:41.358088 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:41.358297 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:41.358903 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:41.359735 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:41.359808 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.359879 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:41.359931 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.368856 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45789
I20260929 18:31:41.368892 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45789 every 8 connection(s)
I20260929 18:31:41.369834 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:41.372292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9267
I20260929 18:31:41.372804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34603
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41865
--webserver_port=35165
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:31:41.381920 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.371599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35443 (local address 127.0.98.62:41865)
0929 18:31:41.371733 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.371737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.372595 (+ 858us) server_negotiation.cc:407] Connection header received
0929 18:31:41.374070 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.374072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.374121 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.374200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.375682 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.376580 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.377708 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.378137 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.378291 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.381754 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35443: BlockingRecv error: recv got EOF from 127.0.98.2:35443 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:31:41.382349 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.371972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:35443)
0929 18:31:41.372436 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.372485 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.373819 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.374386 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.374396 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.374925 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.375520 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.375534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.376720 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.376723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.377583 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.377591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.378299 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.378301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.378441 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.379053 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.379068 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.381559 (+ 2491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":155,"threads_started":1}
W20260929 18:31:41.382931 9396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41865 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41865: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.388656 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.383363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:53575)
0929 18:31:41.383595 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.383607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.383666 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.383911 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.383913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.384036 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.384119 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.384123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.385161 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.385164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.385694 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.385701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.386223 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.386226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.386298 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.386808 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.386827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.388567 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169}
W20260929 18:31:41.388834 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.383481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53575 (local address 127.0.98.62:41865)
0929 18:31:41.383617 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.383623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.383642 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:31:41.383738 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.383741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.383798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.383891 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.384254 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.385053 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.385790 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.386122 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.386219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.388731 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53575: BlockingRecv error: recv got EOF from 127.0.98.2:53575 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.394457 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.389275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:51237)
0929 18:31:41.389452 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.389468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.389547 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.389807 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.389809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.389931 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.389997 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.390001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.391051 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.391054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.391575 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.391583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.392110 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.392113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.392231 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.392702 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.392712 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.394367 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 18:31:41.394624 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.389413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51237 (local address 127.0.98.62:41865)
0929 18:31:41.389512 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.389516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.389530 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:41.389653 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.389656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.389695 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.389782 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.390096 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.390930 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.391681 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.392011 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.392107 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.394512 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51237: BlockingRecv error: recv got EOF from 127.0.98.2:51237 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:31:41.394816 9396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:31:41.484586 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:41.484840 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:41.484881 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:41.488921 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:41.488976 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:41.489063 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:41.494185 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35165
--tserver_master_addrs=127.0.98.62:41865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:41.495167 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:41.496261 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:41.505800 9405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:41.505808 9404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:41.505856 9399 server_base.cc:1034] running on GCE node
W20260929 18:31:41.505808 9407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:41.506459 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:41.507057 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:41.508217 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790706701508199 us; error 28 us; skew 500 ppm
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706701, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:41.511080 9399 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 18:31:41.511935 9399 webserver.cc:460] Webserver started at http://127.0.98.3:35165/ using document root <none> and password file <none>
I20260929 18:31:41.512504 9399 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:41.512573 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:41.516098 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:41.518087 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.519483 9399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:41.519613 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "6ceec59e6dd24b0c8ecc0f4b7ccf3c1e"
format_stamp: "Formatted at 2026-09-29 18:31:40 on dist-test-slave-fmv8"
server_key: "6a7225ed13217a3eb49028ab3ed35b20"
server_key_iv: "268ef8cfce68809995e1b902c282d667"
server_key_version: "encryptionkey@0"
I20260929 18:31:41.520117 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:41.540850 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:41.543869 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:41.544020 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:41.544595 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:41.545341 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:41.545385 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.545423 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:41.545446 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:41.554095 9399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34603
I20260929 18:31:41.554155 9527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34603 every 8 connection(s)
I20260929 18:31:41.555075 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:41.559427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9399
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:31:41.566540 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.556890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58801 (local address 127.0.98.62:41865)
0929 18:31:41.557026 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.557030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.557803 (+ 773us) server_negotiation.cc:407] Connection header received
0929 18:31:41.559120 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.559123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.559180 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.559284 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.560770 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.561291 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.562385 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.562598 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.562699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.566427 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58801: BlockingRecv error: recv got EOF from 127.0.98.3:58801 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:31:41.567165 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.557238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:58801)
0929 18:31:41.557658 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.557698 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.558908 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.559461 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.559468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.559937 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.560634 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.560650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.561409 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.561412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.562257 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.562265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.562746 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.562750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.562920 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.563484 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.563500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.566337 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
W20260929 18:31:41.567651 9528 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41865 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41865: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 18:31:41.569021 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.560420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.0.98.62:41865)
0929 18:31:41.560663 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.560666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.560676 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:31:41.560707 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.560709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.560742 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.560877 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.561574 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.562085 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.562927 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.563152 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.563260 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.564656 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:41.564662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:41.564676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:41.564700 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:41.566414 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:41.566923 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:41.566936 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:41.566940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:41.567053 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:41.567672 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:41.567675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:41.567676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:41.568165 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:41.568374 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:41.568628 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:31:41.568868 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.573071 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.568073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:51149)
0929 18:31:41.568199 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.568213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.568289 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.568553 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.568556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.568714 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.568827 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.568834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.569572 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.569575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.570221 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.570229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.570743 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.570748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.570831 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.571289 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.571300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.572991 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:31:41.573190 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.568165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51149 (local address 127.0.98.62:41865)
0929 18:31:41.568273 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.568277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.568291 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:41.568382 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.568387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.568427 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.568524 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.568914 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.569475 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.570328 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.570625 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.570729 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.573096 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51149: BlockingRecv error: recv got EOF from 127.0.98.3:51149 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:41 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:41.577674 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:41.573679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:35191)
0929 18:31:41.573767 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:41.573780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:41.573843 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:41.574028 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:41.574030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:41.574142 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:41.574201 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.574205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.574821 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.574823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.575301 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:41.575307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.575679 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.575682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:41.575733 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.576072 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:41.576083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:41.577596 (+ 1513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 18:31:41.577800 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:41.573731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35191 (local address 127.0.98.62:41865)
0929 18:31:41.573863 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:41.573866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:41.573886 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:31:41.573912 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:41.573914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:41.573943 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:41.574004 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:41.574286 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.574740 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.575402 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:41.575592 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:41.575665 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:41.577704 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35191: BlockingRecv error: recv got EOF from 127.0.98.3:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 18:31:41.577885 9528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:42.207661 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:42.202087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:37915)
0929 18:31:42.202359 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:42.202378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:42.202482 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:42.202733 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:42.202735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:42.202910 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:42.203022 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.203027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.203787 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.203790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.204484 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.204494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.204969 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.204973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.205069 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.205670 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:42.205683 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:42.207473 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":61,"threads_started":1}
W20260929 18:31:42.207705 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:42.202228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37915 (local address 127.0.98.62:41865)
0929 18:31:42.202447 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:42.202450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:42.202464 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:42.202550 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:42.202552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:42.202592 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:42.202667 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:42.203143 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.203668 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.204616 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.204848 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.204967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.207589 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37915: BlockingRecv error: recv got EOF from 127.0.98.1:37915 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:42.400911 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:42.395443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:47269)
0929 18:31:42.395703 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:42.395716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:42.395809 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:42.396032 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:42.396034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:42.396247 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:42.396339 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.396344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.397024 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.397026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.397595 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.397603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.398025 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.398027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.398087 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.398641 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:42.398658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:42.400797 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
W20260929 18:31:42.401065 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:42.395491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47269 (local address 127.0.98.62:41865)
0929 18:31:42.395632 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:42.395635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:42.395784 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:31:42.395888 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:42.395890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:42.395930 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:42.395996 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:42.396433 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.396940 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.397693 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.397921 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.397996 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.400921 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47269: BlockingRecv error: recv got EOF from 127.0.98.2:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:42 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:42.584236 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:42.578499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:48143)
0929 18:31:42.578781 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:42.578799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:42.578904 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:42.579197 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:42.579200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:42.579396 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:42.579510 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.579516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.580386 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.580394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.580963 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:42.580972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.581385 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.581388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:42.581455 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.582103 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:42.582117 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:42.584023 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
W20260929 18:31:42.584266 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:42.578637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48143 (local address 127.0.98.62:41865)
0929 18:31:42.578806 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:42.578809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:42.578909 (+ 100us) server_negotiation.cc:407] Connection header received
0929 18:31:42.578980 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:42.578983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:42.579030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:42.579110 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:42.579638 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.580256 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.581069 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:42.581289 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:42.581366 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:42.584135 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48143: BlockingRecv error: recv got EOF from 127.0.98.3:48143 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:43.214939 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:43.208576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:54187)
0929 18:31:43.208859 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:43.208875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:43.209016 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:43.209393 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:43.209396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:43.209659 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:43.209752 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.209758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.210558 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.210563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.211350 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.211360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.211849 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.211852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.211937 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.212543 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:43.212555 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:43.214755 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
W20260929 18:31:43.215034 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:43.208674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54187 (local address 127.0.98.62:41865)
0929 18:31:43.208899 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:43.208903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:43.208958 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:31:43.209171 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:43.209176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:43.209236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:43.209359 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:43.209881 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.210438 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.211486 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.211729 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.211848 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.214896 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54187: BlockingRecv error: recv got EOF from 127.0.98.1:54187 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:43.407364 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:43.401817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:43799)
0929 18:31:43.402097 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:43.402110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:43.402206 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:43.402442 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:43.402444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:43.402653 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:43.402749 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.402755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.403523 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.403526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.404131 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.404141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.404646 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.404650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.404736 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.405305 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:43.405317 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:43.407158 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
W20260929 18:31:43.407377 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:43.401948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43799 (local address 127.0.98.62:41865)
0929 18:31:43.402091 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:43.402095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:43.402182 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:31:43.402281 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:43.402283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:43.402331 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:43.402401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:43.402857 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.403391 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.404299 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.404528 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.404642 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.407281 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43799: BlockingRecv error: recv got EOF from 127.0.98.2:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:43 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:43.590437 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:43.584973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:58073)
0929 18:31:43.585284 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:43.585312 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:43.585404 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:43.585735 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:43.585738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:43.585922 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:43.586021 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.586026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.586792 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.586795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.587355 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:43.587362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.587812 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.587815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:43.587909 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.588452 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:43.588466 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:43.590312 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
W20260929 18:31:43.590622 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:43.585220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58073 (local address 127.0.98.62:41865)
0929 18:31:43.585351 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:43.585354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:43.585368 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:43.585554 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:43.585557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:43.585603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:43.585674 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:43.586136 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.586685 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.587463 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:43.587709 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:43.587790 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:43.590458 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58073: BlockingRecv error: recv got EOF from 127.0.98.3:58073 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:44.221349 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:44.215733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:50447)
0929 18:31:44.216002 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:44.216027 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:44.216130 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:44.216404 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:44.216406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:44.216585 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:44.216675 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.216680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.217374 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.217376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.217927 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.217934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.218342 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.218344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.218404 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.218943 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:44.218955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:44.221181 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
W20260929 18:31:44.221578 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:44.215859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50447 (local address 127.0.98.62:41865)
0929 18:31:44.216110 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:44.216114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:44.216127 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:44.216253 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:44.216255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:44.216295 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:44.216368 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:44.216772 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.217284 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.218030 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.218245 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.218321 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.221400 (+ 3079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50447: BlockingRecv error: recv got EOF from 127.0.98.1:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:44.414024 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:44.408202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:56435)
0929 18:31:44.408559 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:44.408572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:44.408705 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:44.409038 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:44.409040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:44.409278 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:44.409428 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.409437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.410328 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.410330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.410908 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.410915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.411352 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.411355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.411411 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.411962 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:44.411976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:44.413877 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
W20260929 18:31:44.414176 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:44.408335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56435 (local address 127.0.98.62:41865)
0929 18:31:44.408491 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:44.408496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:44.408649 (+ 153us) server_negotiation.cc:407] Connection header received
0929 18:31:44.408844 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:44.408847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:44.408903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:44.408998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:44.409553 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.410230 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.411025 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.411247 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.411338 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.414018 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56435: BlockingRecv error: recv got EOF from 127.0.98.2:56435 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:44 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:44.596944 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:44.591287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:40303)
0929 18:31:44.591752 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:44.591767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:44.591917 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:44.592216 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:44.592219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:44.592425 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:44.592510 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.592515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.593257 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.593263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.593828 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:44.593835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.594305 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.594308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:44.594386 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.594933 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:44.594947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:44.596747 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":120,"threads_started":1}
W20260929 18:31:44.596974 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:44.591331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40303 (local address 127.0.98.62:41865)
0929 18:31:44.591496 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:44.591501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:44.591852 (+ 351us) server_negotiation.cc:407] Connection header received
0929 18:31:44.592027 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:44.592031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:44.592087 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:44.592192 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:44.592625 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.593151 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.593945 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:44.594193 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:44.594305 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:44.596874 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40303: BlockingRecv error: recv got EOF from 127.0.98.3:40303 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:45.234157 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:45.222181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:38797)
0929 18:31:45.222498 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:45.222512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:45.222607 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:45.222853 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:45.222855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:45.223053 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:45.223146 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.223152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.223994 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.223997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.224647 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.224657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.225175 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.225189 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.225268 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.231605 (+ 6337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:45.231621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:45.233959 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
W20260929 18:31:45.234230 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:45.222252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38797 (local address 127.0.98.62:41865)
0929 18:31:45.222491 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:45.222495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:45.222579 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:31:45.222685 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:45.222687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:45.222729 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:45.222828 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:45.223252 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.223894 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.224761 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.225050 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.225133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.234093 (+ 8960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38797: BlockingRecv error: recv got EOF from 127.0.98.1:38797 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:45.420544 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:45.414840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:52943)
0929 18:31:45.415235 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:45.415248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:45.415379 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:45.415673 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:45.415676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:45.415873 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:45.415968 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.415974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.416802 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.416808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.417359 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.417366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.417820 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.417823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.417908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.418431 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:45.418443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:45.420428 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":102,"threads_started":1}
W20260929 18:31:45.420734 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:45.414968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52943 (local address 127.0.98.62:41865)
0929 18:31:45.415115 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:45.415119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:45.415330 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:31:45.415504 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:45.415509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:45.415565 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:45.415650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:45.416085 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.416693 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.417482 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.417709 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.417818 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.420603 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52943: BlockingRecv error: recv got EOF from 127.0.98.2:52943 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:45 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:45.603848 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:45.597814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:41877)
0929 18:31:45.598078 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:45.598091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:45.598193 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:45.598634 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:45.598636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:45.598823 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:45.598917 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.598923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.599874 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.599881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.600648 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:45.600659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.601122 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.601125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:45.601196 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.601751 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:45.601763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:45.603710 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
W20260929 18:31:45.604017 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:45.597935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41877 (local address 127.0.98.62:41865)
0929 18:31:45.598111 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:45.598115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:45.598148 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:31:45.598273 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:45.598304 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:45.598351 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:45.598427 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:45.599044 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.599747 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.600788 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:45.601011 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:45.601107 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:45.603863 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41877: BlockingRecv error: recv got EOF from 127.0.98.3:41877 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:46.241015 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:46.235117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:41559)
0929 18:31:46.235392 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:46.235406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:46.235496 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:46.235739 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:46.235741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:46.235923 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:46.236013 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.236017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.236892 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.236895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.237454 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.237462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.237939 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.237942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.238015 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.238525 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:46.238536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:46.240848 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
W20260929 18:31:46.241084 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:46.235177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41559 (local address 127.0.98.62:41865)
0929 18:31:46.235451 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:46.235455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:46.235469 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:46.235577 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:46.235579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:46.235627 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:46.235700 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:46.236123 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.236762 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.237558 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.237788 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.237880 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.240965 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41559: BlockingRecv error: recv got EOF from 127.0.98.1:41559 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:46.427515 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:46.421391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:39229)
0929 18:31:46.421774 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:46.421790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:46.421906 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:46.422196 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:46.422199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:46.422405 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:46.422489 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.422495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.423293 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.423299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.424003 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.424011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.424504 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.424508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.424593 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.425245 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:46.425262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:46.427333 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
W20260929 18:31:46.427546 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:46.421471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39229 (local address 127.0.98.62:41865)
0929 18:31:46.421618 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:46.421625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:46.421861 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:31:46.422035 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:46.422038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:46.422088 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:46.422180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:46.422599 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.423186 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.424113 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.424394 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.424504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.427448 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39229: BlockingRecv error: recv got EOF from 127.0.98.2:39229 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:46 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:46.611481 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:46.604663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:39041)
0929 18:31:46.604991 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:46.605007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:46.605153 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:46.605442 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:46.605445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:46.605710 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:46.605853 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.605864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.607068 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.607071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.607986 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:46.607998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.608605 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.608608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:46.608693 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.609326 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:46.609343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:46.611360 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":119,"threads_started":1}
W20260929 18:31:46.611654 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:46.604761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39041 (local address 127.0.98.62:41865)
0929 18:31:46.604918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:46.604926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:46.605107 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:31:46.605244 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:46.605246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:46.605319 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:46.605391 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:46.605993 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.606961 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.608141 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:46.608490 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:46.608613 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:46.611496 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39041: BlockingRecv error: recv got EOF from 127.0.98.3:39041 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:47.247669 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:47.241918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:46537)
0929 18:31:47.242184 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:47.242204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:47.242300 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:47.242594 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:47.242596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:47.242784 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:47.242884 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.242889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.243648 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.243651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.244325 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.244334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.244785 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.244787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.244846 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.245464 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:47.245476 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:47.247561 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
W20260929 18:31:47.247872 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:47.242071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46537 (local address 127.0.98.62:41865)
0929 18:31:47.242347 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:47.242350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:47.242363 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:31:47.242399 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:47.242401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:47.242443 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:47.242529 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:47.242992 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.243553 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.244455 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.244682 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.244771 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.247700 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46537: BlockingRecv error: recv got EOF from 127.0.98.1:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:47.434134 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:47.428444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:38949)
0929 18:31:47.428714 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:47.428727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:47.428829 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:47.429138 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:47.429140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:47.429326 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:47.429423 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.429428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.430348 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.430350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.430901 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.430910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.431392 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.431394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.431455 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.431968 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:47.431980 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:47.434021 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260929 18:31:47.434440 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:47.428502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38949 (local address 127.0.98.62:41865)
0929 18:31:47.428667 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:47.428672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:47.428805 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:31:47.428916 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:47.428920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:47.428980 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:47.429059 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:47.429535 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.430217 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.431053 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.431275 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.431419 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.434233 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38949: BlockingRecv error: recv got EOF from 127.0.98.2:38949 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:47 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:47.617702 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:47.612285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:60153)
0929 18:31:47.612562 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:47.612575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:47.612680 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:47.612964 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:47.612967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:47.613181 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:47.613268 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.613274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.614032 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.614038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.614600 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:47.614608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.615067 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.615071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:47.615149 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.615681 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:47.615692 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:47.617584 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260929 18:31:47.617892 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:47.612381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60153 (local address 127.0.98.62:41865)
0929 18:31:47.612512 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:47.612519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:47.612652 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:31:47.612793 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:47.612797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:47.612850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:47.612941 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:47.613392 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.613923 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.614737 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:47.614956 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:47.615066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:47.617734 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60153: BlockingRecv error: recv got EOF from 127.0.98.3:60153 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:48.254091 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:48.248492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:49065)
0929 18:31:48.248960 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:48.248973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:48.249082 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:48.249402 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:48.249405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:48.249613 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:48.249735 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.249744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.250490 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.250494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.251058 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.251066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.251513 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.251517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.251601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.252088 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:48.252099 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:48.253926 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":78,"threads_started":1}
W20260929 18:31:48.254171 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:48.248634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49065 (local address 127.0.98.62:41865)
0929 18:31:48.248859 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:48.248863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:48.249068 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:31:48.249225 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:48.249228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:48.249272 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:48.249377 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:48.249841 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.250370 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.251175 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.251401 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.251512 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.254049 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49065: BlockingRecv error: recv got EOF from 127.0.98.1:49065 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":54,"threads_started":1}
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:48.440742 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:48.434895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:49883)
0929 18:31:48.435224 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:48.435237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:48.435363 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:48.435654 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:48.435657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:48.435857 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:48.435947 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.435952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.436771 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.436776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.437357 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.437366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.437819 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.437823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.437908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.438458 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:48.438471 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:48.440543 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":89,"threads_started":1}
W20260929 18:31:48.440814 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:48.434956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49883 (local address 127.0.98.62:41865)
0929 18:31:48.435132 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:48.435137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:48.435312 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:31:48.435468 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:48.435472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:48.435523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:48.435608 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:48.436066 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.436662 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.437479 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.437708 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.437819 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.440659 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49883: BlockingRecv error: recv got EOF from 127.0.98.2:49883 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:48 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:48.624570 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:48.618385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:35685)
0929 18:31:48.618735 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:48.618748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:48.618875 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:48.619170 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:48.619174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:48.619373 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:48.619462 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.619467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.620675 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.620678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.621320 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:48.621330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.621861 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.621864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:48.621957 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.622447 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:48.622458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:48.624401 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
W20260929 18:31:48.624666 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:48.618474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35685 (local address 127.0.98.62:41865)
0929 18:31:48.618645 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:48.618650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:48.618819 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:31:48.618988 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:48.618991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:48.619045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:48.619128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:48.619605 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.620578 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.621427 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:48.621764 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:48.621847 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:48.624556 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35685: BlockingRecv error: recv got EOF from 127.0.98.3:35685 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:49.260314 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:49.254858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:53475)
0929 18:31:49.255163 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:49.255178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:49.255271 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:49.255500 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:49.255503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:49.255689 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:49.255789 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.255794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.256593 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.256596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.257230 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.257240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.257697 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.257700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.257766 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.258379 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:49.258391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:49.260189 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
W20260929 18:31:49.260555 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:49.255044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53475 (local address 127.0.98.62:41865)
0929 18:31:49.255282 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:49.255286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:49.255298 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:31:49.255346 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:49.255348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:49.255386 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:49.255455 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:49.255893 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.256491 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.257340 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.257555 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.257653 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.260366 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53475: BlockingRecv error: recv got EOF from 127.0.98.1:53475 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:49.447328 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:49.441662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:48179)
0929 18:31:49.441917 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:49.441930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:49.442071 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:49.442353 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:49.442356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:49.442563 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:49.442651 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.442657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.443600 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.443606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.444264 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.444273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.444720 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.444724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.444809 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.445305 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:49.445315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:49.447168 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260929 18:31:49.447398 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:49.441715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48179 (local address 127.0.98.62:41865)
0929 18:31:49.441840 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:49.441843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:49.442022 (+ 179us) server_negotiation.cc:407] Connection header received
0929 18:31:49.442175 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:49.442180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:49.442254 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:49.442333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:49.442770 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.443489 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.444379 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.444610 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.444718 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.447267 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48179: BlockingRecv error: recv got EOF from 127.0.98.2:48179 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:49 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:49.630523 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:49.625352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:35121)
0929 18:31:49.625619 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:49.625632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:49.625728 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:49.625964 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:49.625966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:49.626138 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:49.626232 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.626238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.626936 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.626938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.627511 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:49.627519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.627926 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.627928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:49.627990 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.628508 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:49.628539 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:49.630361 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
W20260929 18:31:49.630623 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:49.625470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35121 (local address 127.0.98.62:41865)
0929 18:31:49.625621 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:49.625625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:49.625699 (+ 74us) server_negotiation.cc:407] Connection header received
0929 18:31:49.625813 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:49.625815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:49.625860 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:49.625922 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:49.626333 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.626843 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.627619 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:49.627833 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:49.627911 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:49.630507 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35121: BlockingRecv error: recv got EOF from 127.0.98.3:35121 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:50.271514 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:50.261141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:54373)
0929 18:31:50.261422 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:50.261435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:50.261528 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:50.261771 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:50.261773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:50.261956 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:50.262047 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.262053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.262787 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.262789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.263349 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.263356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.263803 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.263805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.263865 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.269401 (+ 5536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:50.269414 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:50.271346 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
W20260929 18:31:50.271612 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:50.261234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54373 (local address 127.0.98.62:41865)
0929 18:31:50.261467 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:50.261470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:50.261482 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:31:50.261613 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:50.261615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:50.261656 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:50.261735 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:50.262172 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.262696 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.263476 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.263700 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.263789 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.271482 (+ 7693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54373: BlockingRecv error: recv got EOF from 127.0.98.1:54373 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:50.453959 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:50.448078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:49725)
0929 18:31:50.448387 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:50.448410 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:50.448508 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:50.448725 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:50.448727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:50.448958 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:50.449082 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.449090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.450065 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.450068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.450728 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.450740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.451237 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.451239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.451304 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.451791 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:50.451840 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:50.453787 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":141,"threads_started":1}
W20260929 18:31:50.454051 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:50.448218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49725 (local address 127.0.98.62:41865)
0929 18:31:50.448348 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:50.448352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:50.448500 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:31:50.448574 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:50.448576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:50.448623 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:50.448692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:50.449235 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.449936 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.450832 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.451114 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.451195 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.453940 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49725: BlockingRecv error: recv got EOF from 127.0.98.2:49725 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:50 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:50.637265 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:50.631330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:55025)
0929 18:31:50.631641 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:50.631657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:50.631792 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:50.632125 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:50.632128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:50.632355 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:50.632499 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.632511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.633413 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.633416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.634027 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:50.634035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.634600 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.634603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:50.634672 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.635147 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:50.635159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:50.637146 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":121,"threads_started":1}
W20260929 18:31:50.637501 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:50.631507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55025 (local address 127.0.98.62:41865)
0929 18:31:50.631635 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:50.631639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:50.631734 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:31:50.631935 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:50.631939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:50.631993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:50.632105 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:50.632630 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.633177 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.634118 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:50.634458 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:50.634565 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:50.637269 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55025: BlockingRecv error: recv got EOF from 127.0.98.3:55025 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:51.277878 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:51.272294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:52883)
0929 18:31:51.272625 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:51.272639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:51.272729 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:51.273045 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:51.273047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:51.273233 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:51.273334 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.273340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.274091 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.274095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.274661 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.274669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.275171 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.275174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.275246 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.275791 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:51.275802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:51.277756 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260929 18:31:51.278105 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:51.272420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52883 (local address 127.0.98.62:41865)
0929 18:31:51.272647 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:51.272654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:51.272692 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:31:51.272856 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:51.272858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:51.272906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:51.272989 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:51.273442 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.273982 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.274840 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.275062 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.275149 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.277945 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52883: BlockingRecv error: recv got EOF from 127.0.98.1:52883 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:51.460394 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:51.454856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:56393)
0929 18:31:51.455142 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:51.455163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:51.455285 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:51.455557 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:51.455560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:51.455771 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:51.455867 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.455873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.456726 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.456733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.457328 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.457337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.457795 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.457799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.457883 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.458360 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:51.458372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:51.460282 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
W20260929 18:31:51.460527 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:51.454939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56393 (local address 127.0.98.62:41865)
0929 18:31:51.455097 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:51.455102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:51.455237 (+ 135us) server_negotiation.cc:407] Connection header received
0929 18:31:51.455389 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:51.455393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:51.455445 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:51.455533 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:51.455988 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.456617 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.457458 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.457678 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.457787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.460413 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56393: BlockingRecv error: recv got EOF from 127.0.98.2:56393 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:51 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:51.643546 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:51.638031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:56345)
0929 18:31:51.638319 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:51.638337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:51.638437 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:51.638750 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:51.638756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:51.639009 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:51.639132 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.639138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.639848 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.639850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.640468 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:51.640477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.640964 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.640967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:51.641121 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.641615 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:51.641627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:51.643437 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
W20260929 18:31:51.643674 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:51.638169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56345 (local address 127.0.98.62:41865)
0929 18:31:51.638332 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:51.638338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:51.638399 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:31:51.638538 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:51.638540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:51.638587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:51.638748 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:51.639242 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.639759 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.640574 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:51.640829 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:51.640913 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:51.643560 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56345: BlockingRecv error: recv got EOF from 127.0.98.3:56345 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:52.284222 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:52.278720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:36509)
0929 18:31:52.278988 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:52.279001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:52.279086 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:52.279382 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:52.279384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:52.279555 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:52.279649 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.279687 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.280513 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.280516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.281092 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.281100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.281523 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.281526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.281589 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.282091 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:52.282103 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:52.284045 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
W20260929 18:31:52.284348 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:52.278806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36509 (local address 127.0.98.62:41865)
0929 18:31:52.279045 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:52.279048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:52.279062 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:52.279169 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:52.279171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:52.279212 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:52.279369 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:52.279791 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.280414 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.281193 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.281416 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.281491 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.284221 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36509: BlockingRecv error: recv got EOF from 127.0.98.1:36509 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:52.466671 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:52.461176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:54861)
0929 18:31:52.461544 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:52.461556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:52.461696 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:52.461967 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:52.461970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:52.462180 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:52.462271 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.462276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.463034 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.463041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.463611 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.463618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.464097 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.464100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.464222 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.464715 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:52.464727 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:52.466541 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":93,"threads_started":1}
W20260929 18:31:52.466890 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:52.461278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54861 (local address 127.0.98.62:41865)
0929 18:31:52.461492 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:52.461496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:52.461629 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:31:52.461796 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:52.461800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:52.461853 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:52.461939 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:52.462392 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.462922 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.463761 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.463988 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.464096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.466676 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54861: BlockingRecv error: recv got EOF from 127.0.98.2:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:52 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:52.649565 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:52.644354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:58537)
0929 18:31:52.644608 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:52.644625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:52.644725 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:52.644967 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:52.644969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:52.645145 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:52.645238 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.645243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.645968 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.645971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.646528 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:52.646536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.646969 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.646971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:52.647030 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.647527 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:52.647538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:52.649452 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
W20260929 18:31:52.649719 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:52.644407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58537 (local address 127.0.98.62:41865)
0929 18:31:52.644540 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:52.644544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:52.644735 (+ 191us) server_negotiation.cc:407] Connection header received
0929 18:31:52.644802 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:52.644804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:52.644851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:52.644925 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:52.645345 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.645863 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.646643 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:52.646871 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:52.646951 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:52.649589 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58537: BlockingRecv error: recv got EOF from 127.0.98.3:58537 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:53.290932 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:53.284982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:53139)
0929 18:31:53.285290 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:53.285302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:53.285387 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:53.285633 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:53.285635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:53.285832 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:53.285924 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.285929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.286662 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.286665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.287427 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.287435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.288138 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.288174 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.288265 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.288847 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:53.288858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:53.290766 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
W20260929 18:31:53.290994 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:53.285133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53139 (local address 127.0.98.62:41865)
0929 18:31:53.285370 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:53.285374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:53.285388 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:53.285477 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:53.285479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:53.285519 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:53.285605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:53.286026 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.286561 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.287641 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.287976 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.288075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.290878 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53139: BlockingRecv error: recv got EOF from 127.0.98.1:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":52,"threads_started":1}
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:53.473358 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:53.467438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:51517)
0929 18:31:53.467724 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:53.467739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:53.467876 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:53.468162 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:53.468165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:53.468381 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:53.468474 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.468481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.469270 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.469277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.469857 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.469865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.470334 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.470338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.470425 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.471106 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:53.471123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:53.473227 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
W20260929 18:31:53.473493 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:53.467497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51517 (local address 127.0.98.62:41865)
0929 18:31:53.467690 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:53.467695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:53.467825 (+ 130us) server_negotiation.cc:407] Connection header received
0929 18:31:53.467982 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:53.467987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:53.468039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:53.468129 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:53.468605 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.469154 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.469977 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.470222 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.470332 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.473354 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51517: BlockingRecv error: recv got EOF from 127.0.98.2:51517 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:53 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:53.655640 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:53.650425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:45331)
0929 18:31:53.650699 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:53.650711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:53.650810 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:53.651040 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:53.651042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:53.651218 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:53.651310 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.651315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.652025 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.652027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.652619 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:53.652627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.653048 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.653051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:53.653109 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.653587 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:53.653598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:53.655473 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
W20260929 18:31:53.655704 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:53.650479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45331 (local address 127.0.98.62:41865)
0929 18:31:53.650636 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:53.650640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:53.650784 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:31:53.650888 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:53.650890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:53.650935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:53.651006 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:53.651403 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.651930 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.652732 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:53.652952 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:53.653030 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:53.655584 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45331: BlockingRecv error: recv got EOF from 127.0.98.3:45331 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:54.297031 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:54.291723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:40623)
0929 18:31:54.292006 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:54.292019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:54.292104 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:54.292367 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:54.292369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:54.292540 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:54.292639 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.292645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.293381 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.293384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.293943 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.293951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.294408 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.294411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.294472 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.295025 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:54.295037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:54.296920 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
W20260929 18:31:54.297212 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:54.291786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40623 (local address 127.0.98.62:41865)
0929 18:31:54.292049 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:54.292053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:54.292080 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:31:54.292214 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:54.292216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:54.292260 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:54.292332 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:54.292751 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.293284 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.294067 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.294306 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.294394 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.297064 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40623: BlockingRecv error: recv got EOF from 127.0.98.1:40623 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:54.479331 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:54.474048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:56723)
0929 18:31:54.474342 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:54.474358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:54.474491 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:54.474775 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:54.474778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:54.474982 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:54.475072 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.475095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.475849 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.475855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.476479 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.476488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.476933 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.476936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.477026 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.477496 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:54.477507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:54.479215 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
W20260929 18:31:54.479480 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:54.474180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56723 (local address 127.0.98.62:41865)
0929 18:31:54.474322 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:54.474327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:54.474429 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:31:54.474615 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:54.474617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:54.474659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:54.474750 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:54.475199 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.475741 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.476599 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.476816 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.476923 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.479334 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56723: BlockingRecv error: recv got EOF from 127.0.98.2:56723 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:54 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:54.661911 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:54.656554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:58843)
0929 18:31:54.656827 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:54.656843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:54.656953 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:54.657298 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:54.657301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:54.657498 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:54.657595 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.657600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.658340 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.658345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.658914 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:54.658922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.659371 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.659374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:54.659456 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.659981 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:54.659993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:54.661801 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
W20260929 18:31:54.662150 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:54.656609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58843 (local address 127.0.98.62:41865)
0929 18:31:54.656743 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:54.656748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:54.656918 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:31:54.657075 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:54.657077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:54.657154 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:54.657245 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:54.657710 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.658234 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.659044 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:54.659263 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:54.659369 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:54.661926 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58843: BlockingRecv error: recv got EOF from 127.0.98.3:58843 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:55.306859 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:55.297959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:49765)
0929 18:31:55.298254 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:55.298271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:55.298378 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:55.298631 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:55.298632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:55.298804 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:55.298894 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.298899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.299682 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.299685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.300293 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.300301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.300797 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.300800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.300861 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.304603 (+ 3742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:55.304615 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:55.306737 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
W20260929 18:31:55.307018 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:55.298020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49765 (local address 127.0.98.62:41865)
0929 18:31:55.298321 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:55.298325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:55.298339 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:31:55.298482 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:55.298484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:55.298525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:55.298600 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:55.299019 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.299586 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.300456 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.300693 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.300781 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.306887 (+ 6106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49765: BlockingRecv error: recv got EOF from 127.0.98.1:49765 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":70,"threads_started":1}
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:55.485746 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:55.480060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:51037)
0929 18:31:55.480333 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:55.480346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:55.480447 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:55.480690 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:55.480694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:55.480929 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:55.481030 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.481037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.481888 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.481892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.482534 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.482544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.483087 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.483090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.483159 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.483638 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:55.483650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:55.485564 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260929 18:31:55.485781 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:55.480128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51037 (local address 127.0.98.62:41865)
0929 18:31:55.480303 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:55.480307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:55.480398 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:31:55.480522 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:55.480524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:55.480566 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:55.480642 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:55.481133 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.481766 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.482648 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.482939 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.483046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.485678 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51037: BlockingRecv error: recv got EOF from 127.0.98.2:51037 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:55 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:55.668603 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:55.662679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.3:56513)
0929 18:31:55.662943 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:55.662956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:55.663055 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:55.663280 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:55.663283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:55.663461 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:55.663551 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.663556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.664584 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.664590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.665206 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:55.665214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.665719 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.665723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:55.665814 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.666399 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:55.666413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:55.668400 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260929 18:31:55.668634 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:55.662760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56513 (local address 127.0.98.62:41865)
0929 18:31:55.662954 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:55.662958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:55.663029 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:31:55.663134 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:55.663136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:55.663180 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:55.663248 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:55.663719 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.664455 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.665343 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:55.665597 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:55.665715 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:55.668528 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56513: BlockingRecv error: recv got EOF from 127.0.98.3:56513 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 29 18:31:56 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:56 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:56.313894 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:56.307741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.1:58663)
0929 18:31:56.308022 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:56.308043 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:56.308132 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:56.308430 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:56.308432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:56.308613 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:56.308704 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:56.308709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.309784 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.309788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:56.310589 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:56.310600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.311077 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.311079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:56.311142 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:56.311675 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:56.311688 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:56.313770 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20260929 18:31:56.314059 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:56.307881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58663 (local address 127.0.98.62:41865)
0929 18:31:56.308141 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:56.308186 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:56.308205 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:31:56.308263 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:56.308266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:56.308309 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:56.308400 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:56.308808 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.309649 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.310716 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.310969 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.311057 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:56.313918 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58663: BlockingRecv error: recv got EOF from 127.0.98.1:58663 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
Sep 29 18:31:56 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:31:56 dist-test-slave-fmv8 krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:31:56.491995 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:56.486657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41865 (local address 127.0.98.2:58385)
0929 18:31:56.486912 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:56.486927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:56.487022 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:56.487259 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:56.487261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:56.487444 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:56.487534 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:56.487540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.488403 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.488406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:56.488969 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:56.488978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.489419 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.489422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:56.489490 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:56.489973 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:56.489985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:56.491818 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41865: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
W20260929 18:31:56.492049 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:56.486740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58385 (local address 127.0.98.62:41865)
0929 18:31:56.486877 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:56.486881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:56.487002 (+ 121us) server_negotiation.cc:407] Connection header received
0929 18:31:56.487099 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:56.487101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:56.487142 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:56.487210 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:56.487651 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.488261 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.489074 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.489287 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.489372 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:56.491944 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58385: BlockingRecv error: recv got EOF from 127.0.98.2:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:56.563789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9115
I20260929 18:31:56.569734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9267
I20260929 18:31:56.574590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9399
I20260929 18:31:56.579625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9045
2026-09-29T18:31:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17957 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:56Z Disabled control of system clock
I20260929 18:31:56.611848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:31:56.720840 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:56.721107 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:56.721138 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:56.725112 9595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:56.725162 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:56.725178 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:56.725194 9595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:56.725231 9595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:56.729931 9595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:56.730893 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:56.732237 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:56.740315 9603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:56.740424 9595 server_base.cc:1034] running on GCE node
W20260929 18:31:56.740331 9601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:56.740284 9600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:56.741132 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:56.742110 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:56.743259 9595 hybrid_clock.cc:648] HybridClock initialized: now 1790706716743229 us; error 47 us; skew 500 ppm
I20260929 18:31:56.745046 9595 webserver.cc:460] Webserver started at http://127.0.98.62:42671/ using document root <none> and password file <none>
I20260929 18:31:56.745549 9595 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:56.745606 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:56.745756 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:56.747714 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "e4029a804955467ab241237cc5935558"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "5fd6064e48226035d09baeb155d3d20c"
server_key_iv: "be0c12da74f5c631d70126db22ce166b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.748327 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4029a804955467ab241237cc5935558"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "5fd6064e48226035d09baeb155d3d20c"
server_key_iv: "be0c12da74f5c631d70126db22ce166b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.751626 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:31:56.753963 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:56.755115 9595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:31:56.755216 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "e4029a804955467ab241237cc5935558"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "5fd6064e48226035d09baeb155d3d20c"
server_key_iv: "be0c12da74f5c631d70126db22ce166b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.755294 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:56.765214 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:56.765745 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:56.765878 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:56.772612 9595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41397
I20260929 18:31:56.772617 9661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41397 every 8 connection(s)
I20260929 18:31:56.773504 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:56.776492 9662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:56.776980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9595
I20260929 18:31:56.777084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:56.777361 392 external_mini_cluster.cc:1428] Setting key 75fc2c6462084a1ffab1849b7ff9f826
I20260929 18:31:56.782814 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: Bootstrap starting.
I20260929 18:31:56.785302 9662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:56.786175 9662 log.cc:824] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:56.787293 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:56.778701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.0.98.62:41397)
0929 18:31:56.779343 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:56.779360 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:56.779403 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:31:56.780269 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:56.780293 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:56.780759 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:56.781129 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:56.781546 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.782953 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.783817 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:56.784512 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:56.784669 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:56.785419 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:56.785425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:56.785440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:56.785925 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:56.785993 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:56.786019 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:56.786118 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:56.786219 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:31:56.786469 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260929 18:31:56.788041 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: No bootstrap required, opened a new log
I20260929 18:31:56.791520 9662 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 18:31:56.791740 9662 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:56.791821 9662 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4029a804955467ab241237cc5935558, State: Initialized, Role: FOLLOWER
I20260929 18:31:56.792259 9662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 18:31:56.792405 9662 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:56.792469 9662 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:56.792562 9662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:56.793469 9662 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 18:31:56.793776 9662 leader_election.cc:302] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4029a804955467ab241237cc5935558; no voters:
I20260929 18:31:56.794068 9662 leader_election.cc:288] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:56.794175 9667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:56.794430 9667 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [term 1 LEADER]: Becoming Leader. State: Replica: e4029a804955467ab241237cc5935558, State: Running, Role: LEADER
I20260929 18:31:56.794734 9667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 18:31:56.795137 9662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:56.796268 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4029a804955467ab241237cc5935558. Latest consensus state: current_term: 1 leader_uuid: "e4029a804955467ab241237cc5935558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260929 18:31:56.796401 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:56.796700 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4029a804955467ab241237cc5935558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4029a804955467ab241237cc5935558" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260929 18:31:56.796840 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558 [sys.catalog]: This master's current role is: LEADER
W20260929 18:31:56.799418 9682 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:56.799503 9682 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:56.799575 9676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:56.800321 9676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:56.804996 9676 catalog_manager.cc:1347] Generated new cluster ID: ec623a2025c44a44a465bfebe9489e3f
I20260929 18:31:56.805058 9676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:56.818930 9676 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:56.819761 9676 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:56.825824 9676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4029a804955467ab241237cc5935558: Generated new TSK 0
I20260929 18:31:56.826438 9676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:56.838850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:31:56.960227 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:56.960472 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:56.960510 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:56.964485 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:56.964531 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:56.964583 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:56.969164 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:56.970151 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:56.971457 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:56.980651 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:56.980806 9686 server_base.cc:1034] running on GCE node
W20260929 18:31:56.980628 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:56.980618 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:56.981341 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:56.982019 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:56.983232 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790706716983219 us; error 26 us; skew 500 ppm
I20260929 18:31:56.985157 9686 webserver.cc:460] Webserver started at http://127.0.98.1:46137/ using document root <none> and password file <none>
I20260929 18:31:56.985708 9686 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:56.985792 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:56.986011 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:56.988044 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1bc3dc4c15d4a7bbfcc2aa2f7ab984a"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "8ffbb3fc2f97a7de1f837d610ee72e0d"
server_key_iv: "ca1e255de17f0462a376260c4002bc0b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.988689 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1bc3dc4c15d4a7bbfcc2aa2f7ab984a"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "8ffbb3fc2f97a7de1f837d610ee72e0d"
server_key_iv: "ca1e255de17f0462a376260c4002bc0b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.992520 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:31:56.995090 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:56.996567 9686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:31:56.996685 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "e1bc3dc4c15d4a7bbfcc2aa2f7ab984a"
format_stamp: "Formatted at 2026-09-29 18:31:56 on dist-test-slave-fmv8"
server_key: "8ffbb3fc2f97a7de1f837d610ee72e0d"
server_key_iv: "ca1e255de17f0462a376260c4002bc0b"
server_key_version: "encryptionkey@0"
I20260929 18:31:56.996806 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:57.014927 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:57.015621 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:57.015820 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:57.016439 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:57.017357 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:57.017426 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.017493 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:57.017537 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.025710 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37557
I20260929 18:31:57.025782 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37557 every 8 connection(s)
I20260929 18:31:57.026580 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:57.035015 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.028373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46851 (local address 127.0.98.62:41397)
0929 18:31:57.028516 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.028519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.029479 (+ 960us) server_negotiation.cc:407] Connection header received
0929 18:31:57.030800 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.030802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.030845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.030926 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.031739 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.032331 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.033390 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.033611 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.033685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.034634 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:57.034637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:57.034639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:57.034649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:57.034665 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:57.034672 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:57.034728 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:57.034863 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 18:31:57.034907 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:31:57.035637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9686
I20260929 18:31:57.035724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:57.035853 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.028718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.1:46851)
0929 18:31:57.029329 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:57.029369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:57.030613 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:57.031073 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:57.031079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:57.031158 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:57.031621 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.031632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.032432 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.032434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.033286 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.033296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.033736 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.033739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.033889 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.034412 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:57.034414 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:57.034451 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:57.034508 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:57.034738 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:57.034744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:57.034748 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:31:57.034851 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:31:57.035050 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260929 18:31:57.036013 392 external_mini_cluster.cc:1428] Setting key a5d199d605bd8df435a9574b24cd0427
I20260929 18:31:57.037114 9814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 18:31:57.037345 9814 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:57.037889 9814 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 18:31:57.038053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:31:57.039435 9626 ts_manager.cc:194] Registered new tserver with Master: e1bc3dc4c15d4a7bbfcc2aa2f7ab984a (127.0.98.1:37557)
I20260929 18:31:57.041046 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46851
W20260929 18:31:57.149010 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:57.149281 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:57.149319 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:57.153264 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:57.153323 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:57.153378 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:57.158130 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:57.159184 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:57.160617 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:57.170150 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.170161 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.170336 9817 server_base.cc:1034] running on GCE node
W20260929 18:31:57.170449 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.170920 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:57.171551 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:57.172747 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790706717172735 us; error 35 us; skew 500 ppm
I20260929 18:31:57.174686 9817 webserver.cc:460] Webserver started at http://127.0.98.2:35313/ using document root <none> and password file <none>
I20260929 18:31:57.175295 9817 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:57.175385 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:57.175607 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:57.177839 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "1bf0fde923934c4880b34716cfe15f74"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ac647ea79933e4bd8dc7c3efd46d66d2"
server_key_iv: "029e791ae89710dc65c3fde140b40329"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.178372 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1bf0fde923934c4880b34716cfe15f74"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ac647ea79933e4bd8dc7c3efd46d66d2"
server_key_iv: "029e791ae89710dc65c3fde140b40329"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.181984 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:57.184585 9831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.185801 9817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:57.185945 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "1bf0fde923934c4880b34716cfe15f74"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ac647ea79933e4bd8dc7c3efd46d66d2"
server_key_iv: "029e791ae89710dc65c3fde140b40329"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.186058 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:57.200553 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:57.201306 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:57.201501 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:57.202082 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:57.203004 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:57.203073 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.203138 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:57.203182 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.212060 9817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35363
I20260929 18:31:57.212086 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35363 every 8 connection(s)
I20260929 18:31:57.213045 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:57.214663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9817
I20260929 18:31:57.214805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:57.215133 392 external_mini_cluster.cc:1428] Setting key 864e548db319ce97a7ede9c5fe474cf8
I20260929 18:31:57.216990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:31:57.223172 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.214835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48907 (local address 127.0.98.62:41397)
0929 18:31:57.214985 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.214990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.216129 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 18:31:57.217615 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.217618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.217672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.217775 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.219001 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.219755 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.220971 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.221395 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.221522 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.222598 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:57.222604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:57.222608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:57.222627 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:57.222655 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:57.222668 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:57.222756 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:57.222884 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:31:57.222967 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:31:57.223937 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.215272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.2:48907)
0929 18:31:57.215970 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:57.216023 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:57.217405 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:57.217951 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:57.217957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:57.218091 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:57.218827 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.218847 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.219885 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.219887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.220842 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.220852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.221547 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.221549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.221687 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.222334 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:57.222335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:57.222375 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:57.222443 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:57.222753 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:57.222759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:57.222762 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:31:57.222853 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:31:57.223069 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260929 18:31:57.225101 9945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 18:31:57.225342 9945 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:57.225848 9945 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 18:31:57.226832 9626 ts_manager.cc:194] Registered new tserver with Master: 1bf0fde923934c4880b34716cfe15f74 (127.0.98.2:35363)
I20260929 18:31:57.227380 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48907
W20260929 18:31:57.327625 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:57.327862 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:57.327922 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:57.331873 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:57.331923 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:57.332003 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:57.336723 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:57.337710 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:57.338996 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:57.348199 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.348191 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.348229 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.348275 9948 server_base.cc:1034] running on GCE node
I20260929 18:31:57.348730 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:57.349315 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:57.350477 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790706717350446 us; error 43 us; skew 500 ppm
I20260929 18:31:57.352336 9948 webserver.cc:460] Webserver started at http://127.0.98.3:36729/ using document root <none> and password file <none>
I20260929 18:31:57.352872 9948 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:57.352932 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:57.353122 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:57.355014 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a7ce485e42d4453b40f9fe8b5c25704"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a4e79f206d8b1a9e4d74c401b6738499"
server_key_iv: "3a79109571959e7d5e6895a6cfb11e48"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.355563 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a7ce485e42d4453b40f9fe8b5c25704"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a4e79f206d8b1a9e4d74c401b6738499"
server_key_iv: "3a79109571959e7d5e6895a6cfb11e48"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.359113 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:57.361579 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.362756 9948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:57.362882 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "5a7ce485e42d4453b40f9fe8b5c25704"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a4e79f206d8b1a9e4d74c401b6738499"
server_key_iv: "3a79109571959e7d5e6895a6cfb11e48"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.362980 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:57.386857 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:57.387501 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:57.387681 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:57.388288 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:57.389204 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:57.389258 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.389295 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:57.389343 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.398391 9948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46199
I20260929 18:31:57.398422 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46199 every 8 connection(s)
I20260929 18:31:57.399245 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:57.403301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 9948
I20260929 18:31:57.403386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:57.403650 392 external_mini_cluster.cc:1428] Setting key 8ecdb50a47a130b4675eee2b9c59aeb3
I20260929 18:31:57.408357 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.401016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48301 (local address 127.0.98.62:41397)
0929 18:31:57.401170 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.401174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.402274 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 18:31:57.403675 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.403678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.403729 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.403832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.404839 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.405367 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.406556 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.406782 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.406888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.407870 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:57.407874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:57.407877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:57.407893 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:57.407910 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:57.407917 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:57.407986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:57.408164 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:31:57.408220 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:31:57.409117 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.401507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.3:48301)
0929 18:31:57.402109 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:57.402149 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:57.403483 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:57.404006 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:57.404013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:57.404095 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:57.404694 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.404708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.405487 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.405490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.406433 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:57.406443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:57.406918 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:57.406922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:57.407078 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:57.407612 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:57.407614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:57.407650 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:57.407706 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:57.407999 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:57.408010 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:57.408015 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:31:57.408140 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:31:57.408424 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260929 18:31:57.410051 10076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 18:31:57.410239 10076 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:57.410658 10076 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 18:31:57.411516 9626 ts_manager.cc:194] Registered new tserver with Master: 5a7ce485e42d4453b40f9fe8b5c25704 (127.0.98.3:46199)
I20260929 18:31:57.412045 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48301
I20260929 18:31:57.417440 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:31:57.420169 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:57.419236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.0.98.62:41397)
0929 18:31:57.419377 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.419381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.419546 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:31:57.419678 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.419682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.419740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.419825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.420048 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39250: BlockingRecv error: recv got EOF from 127.0.0.1:39250 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:31:57.420450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9686
I20260929 18:31:57.425705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9817
I20260929 18:31:57.430698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9948
I20260929 18:31:57.435710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 9595
2026-09-29T18:31:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:57Z Disabled control of system clock
I20260929 18:31:57.464329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45841
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:31:57.570187 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:57.570425 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:57.570461 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:57.574368 10087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:57.574427 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:57.574450 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:57.574474 10087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:57.574496 10087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:57.579044 10087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:57.580042 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:57.581442 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:57.589671 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.589725 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.589725 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.590023 10087 server_base.cc:1034] running on GCE node
I20260929 18:31:57.590498 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:57.591559 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:57.592742 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790706717592710 us; error 50 us; skew 500 ppm
I20260929 18:31:57.594723 10087 webserver.cc:460] Webserver started at http://127.0.98.62:39259/ using document root <none> and password file <none>
I20260929 18:31:57.595304 10087 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:57.595394 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:57.595609 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:57.597677 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "b5cdf11fbe7141989ba0c27bf495e6cc"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ab1498001c682598852c0b1c12d28a84"
server_key_iv: "acddf66d90d592423a249877ab9cdc67"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.598213 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5cdf11fbe7141989ba0c27bf495e6cc"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ab1498001c682598852c0b1c12d28a84"
server_key_iv: "acddf66d90d592423a249877ab9cdc67"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.601838 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:31:57.604120 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.605573 10087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:31:57.605705 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "b5cdf11fbe7141989ba0c27bf495e6cc"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "ab1498001c682598852c0b1c12d28a84"
server_key_iv: "acddf66d90d592423a249877ab9cdc67"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.605814 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:57.610340 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:57.610906 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:57.611080 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:57.617851 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45841
I20260929 18:31:57.617913 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45841 every 8 connection(s)
I20260929 18:31:57.618742 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:57.619094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10087
I20260929 18:31:57.619227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:57.619570 392 external_mini_cluster.cc:1428] Setting key 813eb22a36420fb2af06213638f8a0ae
I20260929 18:31:57.621685 10154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:57.625334 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.620900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.98.62:45841)
0929 18:31:57.621951 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.621967 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.622001 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:31:57.622625 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.622640 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.622979 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.623256 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.624061 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:57.624066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:57.624078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:57.624088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:57.624136 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:57.624195 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:57.624309 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:57.624422 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:31:57.624667 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":540,"threads_started":1}
I20260929 18:31:57.627840 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc: Bootstrap starting.
I20260929 18:31:57.629974 10154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:57.630800 10154 log.cc:824] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:57.632823 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc: No bootstrap required, opened a new log
I20260929 18:31:57.636447 10154 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } }
I20260929 18:31:57.636744 10154 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:57.636818 10154 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdf11fbe7141989ba0c27bf495e6cc, State: Initialized, Role: FOLLOWER
I20260929 18:31:57.637279 10154 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } }
I20260929 18:31:57.637427 10154 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:57.637509 10154 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:57.637609 10154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:57.638540 10154 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } }
I20260929 18:31:57.638876 10154 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5cdf11fbe7141989ba0c27bf495e6cc; no voters:
I20260929 18:31:57.639184 10154 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:57.639333 10159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:57.639564 10159 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [term 1 LEADER]: Becoming Leader. State: Replica: b5cdf11fbe7141989ba0c27bf495e6cc, State: Running, Role: LEADER
I20260929 18:31:57.639881 10159 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } }
I20260929 18:31:57.640518 10154 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:57.641759 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } } }
I20260929 18:31:57.641899 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:57.641753 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdf11fbe7141989ba0c27bf495e6cc. Latest consensus state: current_term: 1 leader_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdf11fbe7141989ba0c27bf495e6cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45841 } } }
I20260929 18:31:57.642058 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:57.642333 10168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:57.645530 10168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:57.651208 10168 catalog_manager.cc:1347] Generated new cluster ID: 81cb8c12f53448f8bdb2cec32e15ae78
I20260929 18:31:57.651289 10168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:57.658171 10168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:57.658492 10168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:57.672647 10168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5cdf11fbe7141989ba0c27bf495e6cc: Generated new TSK 0
I20260929 18:31:57.673391 10168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:31:57.689131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--builtin_ntp_servers=127.0.98.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:31:57.798969 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:57.799218 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:57.799263 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:57.803387 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:57.803452 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:57.803508 10178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:57.808349 10178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:57.809501 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:57.810901 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:57.820351 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.820358 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.820641 10178 server_base.cc:1034] running on GCE node
W20260929 18:31:57.820395 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.821066 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:57.821714 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:57.822893 10178 hybrid_clock.cc:648] HybridClock initialized: now 1790706717822861 us; error 44 us; skew 500 ppm
I20260929 18:31:57.824754 10178 webserver.cc:460] Webserver started at http://127.0.98.1:45009/ using document root <none> and password file <none>
I20260929 18:31:57.825335 10178 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:57.825420 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:57.825639 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:57.827605 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "d2d2defaf0cc4f32a6b3495a5d482f22"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "73d7b4f427a4532b43065af7213755e1"
server_key_iv: "735b23f31a1d5d1b7c8b9917a6a84dd6"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.828217 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d2defaf0cc4f32a6b3495a5d482f22"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "73d7b4f427a4532b43065af7213755e1"
server_key_iv: "735b23f31a1d5d1b7c8b9917a6a84dd6"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.831655 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:31:57.834192 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.835343 10178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:31:57.835486 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "d2d2defaf0cc4f32a6b3495a5d482f22"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "73d7b4f427a4532b43065af7213755e1"
server_key_iv: "735b23f31a1d5d1b7c8b9917a6a84dd6"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.835594 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:57.840015 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:57.840690 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:57.840858 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:57.841373 10178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:57.842293 10178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:57.842358 10178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.842417 10178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:57.842460 10178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:57.851532 10178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37451
I20260929 18:31:57.851559 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37451 every 8 connection(s)
I20260929 18:31:57.852485 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:57.855055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10178
I20260929 18:31:57.855166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:57.855530 392 external_mini_cluster.cc:1428] Setting key 59fd9ede0d8e7901692c70dd0b1d7fcb
I20260929 18:31:57.857745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--builtin_ntp_servers=127.0.98.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:31:57.858948 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.854197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57675 (local address 127.0.98.62:45841)
0929 18:31:57.854332 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:57.854336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:57.855534 (+ 1198us) server_negotiation.cc:407] Connection header received
0929 18:31:57.857144 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:57.857149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:57.857181 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:57.857240 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:57.858486 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:57.858489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:57.858491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:57.858502 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:57.858520 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:57.858528 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:57.858600 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:57.858770 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:31:57.858816 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:31:57.859820 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:57.854589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45841 (local address 127.0.98.1:57675)
0929 18:31:57.855365 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:57.855425 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:57.856915 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:57.857430 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:57.857439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:57.857554 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:57.858190 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:57.858194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:57.858252 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:57.858339 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:57.858609 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:57.858619 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:57.858628 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:31:57.858807 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:31:57.859042 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260929 18:31:57.860970 10306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45841
I20260929 18:31:57.861197 10306 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:57.861598 10306 heartbeater.cc:511] Master 127.0.98.62:45841 requested a full tablet report, sending...
I20260929 18:31:57.863320 10118 ts_manager.cc:194] Registered new tserver with Master: d2d2defaf0cc4f32a6b3495a5d482f22 (127.0.98.1:37451)
W20260929 18:31:57.966267 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:57.966513 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:57.966550 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:57.970474 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:57.970527 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:57.970579 10309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:57.975059 10309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:57.976008 10309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:57.977375 10309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:57.986495 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.986603 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:57.986685 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:57.986681 10309 server_base.cc:1034] running on GCE node
I20260929 18:31:57.987147 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:57.987735 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:57.988986 10309 hybrid_clock.cc:648] HybridClock initialized: now 1790706717988933 us; error 63 us; skew 500 ppm
I20260929 18:31:57.991000 10309 webserver.cc:460] Webserver started at http://127.0.98.2:42713/ using document root <none> and password file <none>
I20260929 18:31:57.991568 10309 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:57.991657 10309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:57.991881 10309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:57.994017 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "60447f2c685e468e97f8c818f4fc8b7b"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a7224bf340adc77c089f430b0a0dce37"
server_key_iv: "b04db04ffb894d7af9fdcf73ce23a553"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.994607 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60447f2c685e468e97f8c818f4fc8b7b"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a7224bf340adc77c089f430b0a0dce37"
server_key_iv: "b04db04ffb894d7af9fdcf73ce23a553"
server_key_version: "encryptionkey@0"
I20260929 18:31:57.998307 10309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:58.000778 10323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.002002 10309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:58.002128 10309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "60447f2c685e468e97f8c818f4fc8b7b"
format_stamp: "Formatted at 2026-09-29 18:31:57 on dist-test-slave-fmv8"
server_key: "a7224bf340adc77c089f430b0a0dce37"
server_key_iv: "b04db04ffb894d7af9fdcf73ce23a553"
server_key_version: "encryptionkey@0"
I20260929 18:31:58.002238 10309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:58.006785 10309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:58.007375 10309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:58.007555 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:58.008080 10309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:58.009006 10309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:58.009075 10309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.009140 10309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:58.009183 10309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.017900 10309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46017
I20260929 18:31:58.017923 10436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46017 every 8 connection(s)
I20260929 18:31:58.018736 10309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:31:58.024415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10309
I20260929 18:31:58.024498 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:58.020482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35281 (local address 127.0.98.62:45841)
0929 18:31:58.020615 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:58.020618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:58.021636 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 18:31:58.022859 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:58.022861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:58.022897 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:58.022972 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:58.024017 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:58.024019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:58.024021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:58.024029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:58.024046 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:58.024053 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:58.024123 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:58.024337 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:31:58.024372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:31:58.024539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:31:58.024827 392 external_mini_cluster.cc:1428] Setting key 8d0861d96a87ed5622b569212027e41d
I20260929 18:31:58.025501 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:58.020872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45841 (local address 127.0.98.2:35281)
0929 18:31:58.021497 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:58.021536 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:58.022662 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:58.023120 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:58.023126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:58.023210 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:58.023791 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:58.023792 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:58.023830 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:58.023894 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:58.024139 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:58.024150 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:58.024176 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:31:58.024391 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 18:31:58.024634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260929 18:31:58.026510 10437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45841
I20260929 18:31:58.026737 10437 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:58.026762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--builtin_ntp_servers=127.0.98.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:31:58.027144 10437 heartbeater.cc:511] Master 127.0.98.62:45841 requested a full tablet report, sending...
I20260929 18:31:58.028170 10117 ts_manager.cc:194] Registered new tserver with Master: 60447f2c685e468e97f8c818f4fc8b7b (127.0.98.2:46017)
W20260929 18:31:58.133525 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:58.133770 10440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:58.133808 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:58.137704 10440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:58.137759 10440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:58.137812 10440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:31:58.142690 10440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:58.143618 10440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:58.144948 10440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:58.154070 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:58.154049 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:58.154152 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:58.154232 10440 server_base.cc:1034] running on GCE node
I20260929 18:31:58.154714 10440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:58.155303 10440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:58.156454 10440 hybrid_clock.cc:648] HybridClock initialized: now 1790706718156439 us; error 26 us; skew 500 ppm
I20260929 18:31:58.158226 10440 webserver.cc:460] Webserver started at http://127.0.98.3:38971/ using document root <none> and password file <none>
I20260929 18:31:58.158739 10440 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:58.158798 10440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:58.158957 10440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:58.161092 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a1999f4c3ff41b58f84c0ffff82a011"
format_stamp: "Formatted at 2026-09-29 18:31:58 on dist-test-slave-fmv8"
server_key: "f1bb802e7ea9ae3b259aefe17caffc63"
server_key_iv: "74367a2f56785fe8e91ca971171ce774"
server_key_version: "encryptionkey@0"
I20260929 18:31:58.161578 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a1999f4c3ff41b58f84c0ffff82a011"
format_stamp: "Formatted at 2026-09-29 18:31:58 on dist-test-slave-fmv8"
server_key: "f1bb802e7ea9ae3b259aefe17caffc63"
server_key_iv: "74367a2f56785fe8e91ca971171ce774"
server_key_version: "encryptionkey@0"
I20260929 18:31:58.165020 10440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:31:58.167287 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.168541 10440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:58.168640 10440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "2a1999f4c3ff41b58f84c0ffff82a011"
format_stamp: "Formatted at 2026-09-29 18:31:58 on dist-test-slave-fmv8"
server_key: "f1bb802e7ea9ae3b259aefe17caffc63"
server_key_iv: "74367a2f56785fe8e91ca971171ce774"
server_key_version: "encryptionkey@0"
I20260929 18:31:58.168721 10440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:58.173043 10440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:58.173543 10440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:58.173681 10440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:58.174177 10440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:58.175019 10440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:58.175064 10440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.175100 10440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:58.175122 10440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:58.183493 10440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43655
I20260929 18:31:58.183522 10567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43655 every 8 connection(s)
I20260929 18:31:58.184388 10440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:31:58.189975 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:58.185994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57797 (local address 127.0.98.62:45841)
0929 18:31:58.186146 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:58.186150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:58.187150 (+ 1000us) server_negotiation.cc:407] Connection header received
0929 18:31:58.188489 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:58.188492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:58.188527 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:58.188592 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:58.189640 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:58.189643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:58.189645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:31:58.189654 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:58.189671 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:31:58.189678 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:58.189747 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:58.189837 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:31:58.189875 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:31:58.190766 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:58.186376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45841 (local address 127.0.98.3:57797)
0929 18:31:58.187012 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:58.187050 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:58.188292 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:58.188756 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:58.188762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:58.188846 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:58.189413 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:31:58.189415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:58.189453 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:31:58.189517 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:58.189735 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:58.189739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:58.189743 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:31:58.189860 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:31:58.190037 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260929 18:31:58.191732 10568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45841
I20260929 18:31:58.191927 10568 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:58.192373 10568 heartbeater.cc:511] Master 127.0.98.62:45841 requested a full tablet report, sending...
I20260929 18:31:58.192444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10440
I20260929 18:31:58.192567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:31:58.192834 392 external_mini_cluster.cc:1428] Setting key db91aa04548384110fb0c5cb5685d649
I20260929 18:31:58.193403 10117 ts_manager.cc:194] Registered new tserver with Master: 2a1999f4c3ff41b58f84c0ffff82a011 (127.0.98.3:43655)
I20260929 18:31:58.194681 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:31:58.197517 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:31:58.196668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.0.98.62:45841)
0929 18:31:58.196782 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:58.196788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:58.196968 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:31:58.197104 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:58.197107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:58.197142 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:58.197218 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:58.197430 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36560: BlockingRecv error: recv got EOF from 127.0.0.1:36560 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:31:58.197894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10178
I20260929 18:31:58.203485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10309
I20260929 18:31:58.208586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10440
I20260929 18:31:58.213420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10087
2026-09-29T18:31:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (777 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:58 dist-test-slave-fmv8 krb5kdc[10578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:58 dist-test-slave-fmv8 krb5kdc[10578](info): set up 2 sockets
Sep 29 18:31:58 dist-test-slave-fmv8 krb5kdc[10578](info): commencing operation
krb5kdc: starting...
W20260929 18:31:59.257076 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:31:59.371153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:59.478564 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:59.478797 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:59.478835 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:59.482807 10591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:59.482867 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:59.482887 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:59.482903 10591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:59.482918 10591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:59.488087 10591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:59.489058 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:59.490051 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:59.498234 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:59.498297 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.498324 10591 server_base.cc:1034] running on GCE node
W20260929 18:31:59.498483 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.498991 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:59.499963 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:59.501127 10591 hybrid_clock.cc:648] HybridClock initialized: now 1790706719501132 us; error 46 us; skew 500 ppm
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:59.503767 10591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:31:59.504606 10591 webserver.cc:460] Webserver started at http://127.0.98.62:42709/ using document root <none> and password file <none>
I20260929 18:31:59.505084 10591 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:59.505139 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:59.505295 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:59.507182 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "588519492e2044cca663454b2c1904b7"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "f159536485ba1d42092813ad923888c7"
server_key_iv: "4d14317155f17ad66c8caa932e0891c3"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.507668 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "588519492e2044cca663454b2c1904b7"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "f159536485ba1d42092813ad923888c7"
server_key_iv: "4d14317155f17ad66c8caa932e0891c3"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.511099 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:59.513319 10606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.514461 10591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:59.514607 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "588519492e2044cca663454b2c1904b7"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "f159536485ba1d42092813ad923888c7"
server_key_iv: "4d14317155f17ad66c8caa932e0891c3"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.514708 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:59.526325 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:59.529175 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:59.529320 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:59.535702 10591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41143
I20260929 18:31:59.535971 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41143 every 8 connection(s)
I20260929 18:31:59.536687 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:59.539276 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:59.544410 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: Bootstrap starting.
I20260929 18:31:59.546716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10591
I20260929 18:31:59.546721 10659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:59.546840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:31:59.547123 392 external_mini_cluster.cc:1428] Setting key db73794eaf90376823023987b812a2ed
I20260929 18:31:59.547580 10659 log.cc:824] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:59.549738 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: No bootstrap required, opened a new log
I20260929 18:31:59.553915 10659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } }
I20260929 18:31:59.554198 10659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:59.554287 10659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588519492e2044cca663454b2c1904b7, State: Initialized, Role: FOLLOWER
I20260929 18:31:59.554714 10659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } }
I20260929 18:31:59.554873 10659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:59.554962 10659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:59.555065 10659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:59.556334 10659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } }
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:59.556660 10659 leader_election.cc:302] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 588519492e2044cca663454b2c1904b7; no voters:
I20260929 18:31:59.556944 10659 leader_election.cc:288] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:59.557076 10664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:59.557304 10664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [term 1 LEADER]: Becoming Leader. State: Replica: 588519492e2044cca663454b2c1904b7, State: Running, Role: LEADER
I20260929 18:31:59.557693 10664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } }
I20260929 18:31:59.558171 10659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:59.559553 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "588519492e2044cca663454b2c1904b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } } }
I20260929 18:31:59.559703 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:59.559558 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588519492e2044cca663454b2c1904b7. Latest consensus state: current_term: 1 leader_uuid: "588519492e2044cca663454b2c1904b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "588519492e2044cca663454b2c1904b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41143 } } }
I20260929 18:31:59.560088 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:59.562915 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:59.548292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.0.98.62:41143)
0929 18:31:59.548675 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:59.548692 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:59.548733 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:31:59.549528 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:59.549550 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:59.549939 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:59.550291 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:59.550977 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.552503 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.553415 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.554091 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.554273 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:59.557400 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:59.557406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:59.557425 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:59.557456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:59.559582 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.560131 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.560140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.560175 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.560321 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.560655 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.560659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.560660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.561073 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:59.561264 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:59.561745 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 18:31:59.562023 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":91,"threads_started":1}
W20260929 18:31:59.563206 10679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:59.563287 10679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:59.563360 10675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:59.564019 10675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:59.569409 10675 catalog_manager.cc:1347] Generated new cluster ID: 96c83b8b06c24e0a970cc2281509e28e
I20260929 18:31:59.569468 10675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:59.581022 10675 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:59.581888 10675 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:59.586908 10675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: Generated new TSK 0
I20260929 18:31:59.587512 10675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:31:59.620294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:59.728711 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:59.728962 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:59.729043 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:59.732976 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:59.733042 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:59.733119 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:31:59.738173 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:59.739095 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:59.740029 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:59.749166 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.749253 10685 server_base.cc:1034] running on GCE node
W20260929 18:31:59.749159 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:59.749159 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.749722 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:59.750331 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:59.751473 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790706719751462 us; error 27 us; skew 500 ppm
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:59.754060 10685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:31:59.754869 10685 webserver.cc:460] Webserver started at http://127.0.98.1:45513/ using document root <none> and password file <none>
I20260929 18:31:59.755394 10685 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:59.755479 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:59.755685 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:59.757792 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "796681cc50e74a5788883c41b9a74d06"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "c20bf8a4ea6a7a0c83da116b98fdffc5"
server_key_iv: "45a6b20a23fba2f062a739ee703e1f00"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.758417 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "796681cc50e74a5788883c41b9a74d06"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "c20bf8a4ea6a7a0c83da116b98fdffc5"
server_key_iv: "45a6b20a23fba2f062a739ee703e1f00"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.762133 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:31:59.764521 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.765704 10685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:31:59.765826 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "796681cc50e74a5788883c41b9a74d06"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "c20bf8a4ea6a7a0c83da116b98fdffc5"
server_key_iv: "45a6b20a23fba2f062a739ee703e1f00"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.765925 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:59.785420 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:59.788626 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:59.788818 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:59.789407 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:59.790196 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:59.790248 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.790315 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:59.790344 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.798905 10685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39925
I20260929 18:31:59.798930 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39925 every 8 connection(s)
I20260929 18:31:59.799779 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:31:59.805781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10685
I20260929 18:31:59.805876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:31:59.806143 392 external_mini_cluster.cc:1428] Setting key e821d28ec0405026a9f03b41b2d7d5ef
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:31:59.814188 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:59.801492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49539 (local address 127.0.98.62:41143)
0929 18:31:59.801638 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:59.801642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:59.802423 (+ 781us) server_negotiation.cc:407] Connection header received
0929 18:31:59.803660 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:59.803664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:59.803721 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:59.803817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:59.805026 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.805603 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.806960 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.807290 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.807413 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:59.810431 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:59.810436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:59.810443 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:59.810468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:59.811891 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.812512 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.812515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.812516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.812600 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.813030 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.813032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.813033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.813199 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:59.813271 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:59.813964 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 18:31:59.814064 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:31:59.815351 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:59.801872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41143 (local address 127.0.98.1:49539)
0929 18:31:59.802283 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 18:31:59.802322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:31:59.803453 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:31:59.803986 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:31:59.803992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:31:59.804431 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:31:59.804880 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:59.804893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.805779 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.805782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:59.806851 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:31:59.806860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.807468 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.807471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:31:59.807682 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:59.808337 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:31:59.808363 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:31:59.810264 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:31:59.812003 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:59.812012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:59.812028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:59.812418 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:59.812713 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:31:59.812717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:31:59.812721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:31:59.812926 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:31:59.813302 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:31:59.813312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:31:59.813686 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 18:31:59.813914 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 18:31:59.814282 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260929 18:31:59.816522 10814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41143
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:31:59.816776 10814 heartbeater.cc:464] Registering TS with master...
I20260929 18:31:59.817238 10814 heartbeater.cc:511] Master 127.0.98.62:41143 requested a full tablet report, sending...
I20260929 18:31:59.818742 10623 ts_manager.cc:194] Registered new tserver with Master: 796681cc50e74a5788883c41b9a74d06 (127.0.98.1:39925)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:31:59.820230 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:31:59.834652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:59.941083 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:59.941329 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:59.941370 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:59.945472 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:59.945533 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:59.945608 10819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:31:59.950691 10819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:31:59.951598 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:59.952576 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:59.961994 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:59.962064 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.962153 10819 server_base.cc:1034] running on GCE node
W20260929 18:31:59.962065 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.962584 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:59.963238 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:59.964438 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790706719964420 us; error 30 us; skew 500 ppm
Sep 29 18:31:59 dist-test-slave-fmv8 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:59.967084 10819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:31:59.967840 10819 webserver.cc:460] Webserver started at http://127.0.98.2:35521/ using document root <none> and password file <none>
I20260929 18:31:59.968442 10819 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:59.968510 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:59.968668 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:59.971050 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc25d37ae32c422db047945e8b4aa443"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "df94c76cd7736eb2d6ac3b3b1230c678"
server_key_iv: "d3c3dd2758a5d65eccd27f3ed34c8b22"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.971598 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc25d37ae32c422db047945e8b4aa443"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "df94c76cd7736eb2d6ac3b3b1230c678"
server_key_iv: "d3c3dd2758a5d65eccd27f3ed34c8b22"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.975143 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:31:59.977475 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.978735 10819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:31:59.978875 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "fc25d37ae32c422db047945e8b4aa443"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-fmv8"
server_key: "df94c76cd7736eb2d6ac3b3b1230c678"
server_key_iv: "d3c3dd2758a5d65eccd27f3ed34c8b22"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.978984 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:59.990082 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:59.993221 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:59.993417 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:59.993993 10819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:31:59.994789 10819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:31:59.994856 10819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.994926 10819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:31:59.994966 10819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.004216 10819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43857
I20260929 18:32:00.004245 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43857 every 8 connection(s)
I20260929 18:32:00.005146 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:00.010299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10819
I20260929 18:32:00.010392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:00.010677 392 external_mini_cluster.cc:1428] Setting key f5beed46fd594498fc861111381aec52
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:00.019779 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.007027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48919 (local address 127.0.98.62:41143)
0929 18:32:00.007170 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.007174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.008463 (+ 1289us) server_negotiation.cc:407] Connection header received
0929 18:32:00.009397 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.009400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.009457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.009565 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.010807 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.011463 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.012717 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.013125 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.013248 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.015551 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.015556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.015559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.015592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.017439 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.017973 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.017979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.017982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.018092 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.018483 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.018486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.018488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.018727 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.018832 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.019460 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.019586 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:32:00.020581 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.007380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41143 (local address 127.0.98.2:48919)
0929 18:32:00.008318 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.008361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.009194 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.009733 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.009740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.010118 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:00.010641 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.010655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.011581 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.011583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.012599 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.012608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.013259 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.013262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.013412 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.014022 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:00.014038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:00.015395 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:00.017591 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.017601 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.017613 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.017874 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.018230 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.018233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.018235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.018383 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.018843 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:00.018848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:00.019119 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.019370 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.019623 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":534,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:32:00.021703 10948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41143
I20260929 18:32:00.021946 10948 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:00.022389 10948 heartbeater.cc:511] Master 127.0.98.62:41143 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:32:00.023412 10623 ts_manager.cc:194] Registered new tserver with Master: fc25d37ae32c422db047945e8b4aa443 (127.0.98.2:43857)
I20260929 18:32:00.024258 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:32:00.037369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:00.153017 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:00.153262 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:00.153306 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:00.157315 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:00.157373 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:00.157467 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:32:00.162930 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:32:00.164036 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:00.165067 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:00.174574 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.174559 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.174566 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:00.174633 10953 server_base.cc:1034] running on GCE node
I20260929 18:32:00.175333 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:00.175988 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:00.177172 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790706720177155 us; error 38 us; skew 500 ppm
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:00.179854 10953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:32:00.180806 10953 webserver.cc:460] Webserver started at http://127.0.98.3:36249/ using document root <none> and password file <none>
I20260929 18:32:00.181367 10953 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:00.181455 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:00.181684 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:00.183823 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "14ff559b20e84d99ad466dcfdec6c361"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-fmv8"
server_key: "e9f3bcac989e0df766579c75a745839f"
server_key_iv: "6f6aad9baeba446b2e4c3fbd400a22b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.184406 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14ff559b20e84d99ad466dcfdec6c361"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-fmv8"
server_key: "e9f3bcac989e0df766579c75a745839f"
server_key_iv: "6f6aad9baeba446b2e4c3fbd400a22b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.188036 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:00.190539 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.191954 10953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:00.192087 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "14ff559b20e84d99ad466dcfdec6c361"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-fmv8"
server_key: "e9f3bcac989e0df766579c75a745839f"
server_key_iv: "6f6aad9baeba446b2e4c3fbd400a22b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.192234 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:00.213112 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:00.216279 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:00.216485 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:00.217103 10953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:00.218007 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:00.218075 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.218150 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:00.218194 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.227257 10953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34193
I20260929 18:32:00.227279 11081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34193 every 8 connection(s)
I20260929 18:32:00.228142 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:00.233695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 10953
I20260929 18:32:00.233812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:00.234083 392 external_mini_cluster.cc:1428] Setting key c3d99686b2b427dd4c7db65f8d6fa9b5
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:00.241972 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.229861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35829 (local address 127.0.98.62:41143)
0929 18:32:00.230008 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.230013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.230879 (+ 866us) server_negotiation.cc:407] Connection header received
0929 18:32:00.232126 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.232129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.232234 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.232315 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.233551 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.234488 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.235866 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.236127 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.236255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.238754 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.238757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.238759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.238783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.240019 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.240567 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.240570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.240572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.240651 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.240985 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.240987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.240988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.241164 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.241240 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.241743 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.241843 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:32:00.242745 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.230291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41143 (local address 127.0.98.3:35829)
0929 18:32:00.230722 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.230765 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.231906 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.232474 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.232482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.232866 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:00.233392 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.233405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.234654 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.234657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.235742 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.235752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.236295 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.236300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.236510 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.237104 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:00.237121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:00.238594 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:00.240129 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.240133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.240145 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.240465 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.240762 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.240764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.240766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.240901 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.241275 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:00.241280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:00.241510 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.241667 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.241906 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260929 18:32:00.243777 11082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41143
I20260929 18:32:00.244055 11082 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:00.244563 11082 heartbeater.cc:511] Master 127.0.98.62:41143 requested a full tablet report, sending...
I20260929 18:32:00.245580 10623 ts_manager.cc:194] Registered new tserver with Master: 14ff559b20e84d99ad466dcfdec6c361 (127.0.98.3:34193)
I20260929 18:32:00.246109 10623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35829
I20260929 18:32:00.247917 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:00.256484 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.249748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.0.98.62:41143)
0929 18:32:00.249863 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.249866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.250057 (+ 191us) server_negotiation.cc:407] Connection header received
0929 18:32:00.250166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.250168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.250204 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.250269 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.250874 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.251367 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.252139 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.252395 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.252473 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.253740 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.253743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.253745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.253766 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.254990 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.255358 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.255360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.255361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.255439 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.255683 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.255686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.255687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.255912 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.256046 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.256278 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.256370 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:32:00.259532 10623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:00.262697 10623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:00.276374 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.270916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43857 (local address 127.0.0.1:54466)
0929 18:32:00.271372 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.271418 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.271551 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.272381 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.272387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.272410 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:00.272567 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.272574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.273962 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.273969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.274877 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.274888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.275819 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.275823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.275924 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.275957 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.276095 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.276242 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":83,"threads_started":1}
I20260929 18:32:00.276386 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.271086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.0.98.2:43857)
0929 18:32:00.271846 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.271858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.271882 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:32:00.271963 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.271968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.272207 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.272725 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:00.272888 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.273846 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.275017 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.275702 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.275821 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.275966 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.276125 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.276237 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":113,"threads_started":1}
I20260929 18:32:00.277199 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.271015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34193 (local address 127.0.0.1:34650)
0929 18:32:00.271636 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.271649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.271738 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.273004 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.273007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.273024 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:00.273118 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.273124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.274381 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.274384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.275561 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.275571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.276785 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.276789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.276874 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.276896 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.276967 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.277045 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":23,"thread_start_us":75,"threads_started":1}
I20260929 18:32:00.277599 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.272046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.3:34193)
0929 18:32:00.272608 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.272624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.272648 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:32:00.272718 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.272723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.272883 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.273036 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:00.273206 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.274238 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.275691 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.276518 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.276612 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.276729 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.277379 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.277468 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":169,"threads_started":1}
I20260929 18:32:00.277628 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.271836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.1:39925)
0929 18:32:00.272490 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.272506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.272813 (+ 307us) server_negotiation.cc:407] Connection header received
0929 18:32:00.272889 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.272894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.273080 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.273220 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:00.273573 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.274654 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.276289 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.277074 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.277172 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.277316 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.277421 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.277500 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":91,"threads_started":1}
I20260929 18:32:00.277592 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.271637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.0.1:42056)
0929 18:32:00.272635 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.272650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.272729 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.273359 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.273362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.273377 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:00.273451 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.273456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.274940 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.274944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.276134 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.276167 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.277188 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.277192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.277286 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.277312 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.277400 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.277464 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260929 18:32:00.279462 10882 tablet_service.cc:1467] Processing CreateTablet for tablet 790cfecf72db4fb686045a0fd1e05919 (DEFAULT_TABLE table=test-table [id=fa432e5be506488c96d0a8e12393a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:00.280583 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790cfecf72db4fb686045a0fd1e05919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:00.280474 11016 tablet_service.cc:1467] Processing CreateTablet for tablet 790cfecf72db4fb686045a0fd1e05919 (DEFAULT_TABLE table=test-table [id=fa432e5be506488c96d0a8e12393a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:00.280478 10748 tablet_service.cc:1467] Processing CreateTablet for tablet 790cfecf72db4fb686045a0fd1e05919 (DEFAULT_TABLE table=test-table [id=fa432e5be506488c96d0a8e12393a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:00.281437 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790cfecf72db4fb686045a0fd1e05919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:00.281443 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790cfecf72db4fb686045a0fd1e05919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:00.286408 11099 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Bootstrap starting.
I20260929 18:32:00.288638 11101 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: Bootstrap starting.
I20260929 18:32:00.289021 11100 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Bootstrap starting.
I20260929 18:32:00.289309 11099 tablet_bootstrap.cc:654] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:00.289978 11099 log.cc:824] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:00.290652 11101 tablet_bootstrap.cc:654] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:00.291302 11100 tablet_bootstrap.cc:654] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:00.291694 11099 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: No bootstrap required, opened a new log
I20260929 18:32:00.291878 11099 ts_tablet_manager.cc:1397] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:32:00.291878 11101 log.cc:824] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:00.292001 11100 log.cc:824] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:00.293854 11100 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: No bootstrap required, opened a new log
I20260929 18:32:00.294054 11100 ts_tablet_manager.cc:1397] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:32:00.294973 11099 raft_consensus.cc:348] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.295193 11099 raft_consensus.cc:374] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:00.295270 11099 raft_consensus.cc:729] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc25d37ae32c422db047945e8b4aa443, State: Initialized, Role: FOLLOWER
I20260929 18:32:00.295701 11099 consensus_queue.cc:260] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.296368 10948 heartbeater.cc:503] Master 127.0.98.62:41143 was elected leader, sending a full tablet report...
I20260929 18:32:00.296685 11099 ts_tablet_manager.cc:1428] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:32:00.297596 11100 raft_consensus.cc:348] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.298013 11100 raft_consensus.cc:374] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:00.298216 11100 raft_consensus.cc:729] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ff559b20e84d99ad466dcfdec6c361, State: Initialized, Role: FOLLOWER
I20260929 18:32:00.298379 11101 tablet_bootstrap.cc:492] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: No bootstrap required, opened a new log
I20260929 18:32:00.298609 11101 ts_tablet_manager.cc:1397] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:32:00.298774 11100 consensus_queue.cc:260] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.299430 11082 heartbeater.cc:503] Master 127.0.98.62:41143 was elected leader, sending a full tablet report...
I20260929 18:32:00.299748 11100 ts_tablet_manager.cc:1428] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:32:00.302471 11101 raft_consensus.cc:348] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.302783 11101 raft_consensus.cc:374] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:00.302845 11101 raft_consensus.cc:729] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796681cc50e74a5788883c41b9a74d06, State: Initialized, Role: FOLLOWER
I20260929 18:32:00.303231 11101 consensus_queue.cc:260] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.303814 10814 heartbeater.cc:503] Master 127.0.98.62:41143 was elected leader, sending a full tablet report...
I20260929 18:32:00.304095 11101 ts_tablet_manager.cc:1428] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:00.330999 11106 raft_consensus.cc:482] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:00.331210 11106 raft_consensus.cc:504] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.332304 11106 leader_election.cc:288] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc25d37ae32c422db047945e8b4aa443 (127.0.98.2:43857), 796681cc50e74a5788883c41b9a74d06 (127.0.98.1:39925)
I20260929 18:32:00.337306 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.332684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54917 (local address 127.0.98.2:43857)
0929 18:32:00.332806 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.332810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.332825 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:00.332972 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.332975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.333027 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.333096 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:00.333399 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.334196 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.335898 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.336551 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.336654 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.336715 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.337106 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.337181 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:32:00.337306 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.332764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56139 (local address 127.0.98.1:39925)
0929 18:32:00.332886 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.332893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.333166 (+ 273us) server_negotiation.cc:407] Connection header received
0929 18:32:00.333256 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.333258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.333297 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.333379 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:00.333596 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.334288 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.335897 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.336551 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.336654 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.336715 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.337106 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.337181 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:32:00.337335 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.332559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43857 (local address 127.0.98.3:54917)
0929 18:32:00.332710 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.332727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.332830 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.333128 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.333134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.333155 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:00.333291 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.333296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.334302 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.334304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.335771 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.335782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.336777 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.336781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.336929 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.337003 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.337076 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.337177 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":160}
I20260929 18:32:00.337348 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.332559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.98.3:56139)
0929 18:32:00.333047 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.333064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.333164 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.333389 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.333394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.333410 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:00.333503 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.333506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.334381 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.334384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.335771 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.335782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.336777 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.336781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.336934 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.336978 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.337064 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.337177 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":267,"thread_start_us":108,"threads_started":1}
I20260929 18:32:00.337951 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790cfecf72db4fb686045a0fd1e05919" candidate_uuid: "14ff559b20e84d99ad466dcfdec6c361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc25d37ae32c422db047945e8b4aa443" is_pre_election: true
I20260929 18:32:00.338044 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790cfecf72db4fb686045a0fd1e05919" candidate_uuid: "14ff559b20e84d99ad466dcfdec6c361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796681cc50e74a5788883c41b9a74d06" is_pre_election: true
I20260929 18:32:00.338284 10902 raft_consensus.cc:2454] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ff559b20e84d99ad466dcfdec6c361 in term 0.
I20260929 18:32:00.338317 10768 raft_consensus.cc:2454] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ff559b20e84d99ad466dcfdec6c361 in term 0.
I20260929 18:32:00.338760 10972 leader_election.cc:302] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14ff559b20e84d99ad466dcfdec6c361, 796681cc50e74a5788883c41b9a74d06, fc25d37ae32c422db047945e8b4aa443; no voters:
I20260929 18:32:00.339044 11106 raft_consensus.cc:2781] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:00.339123 11106 raft_consensus.cc:482] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:00.339152 11106 raft_consensus.cc:3037] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:00.340176 11106 raft_consensus.cc:504] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.340521 11106 leader_election.cc:288] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [CANDIDATE]: Term 1 election: Requested vote from peers fc25d37ae32c422db047945e8b4aa443 (127.0.98.2:43857), 796681cc50e74a5788883c41b9a74d06 (127.0.98.1:39925)
I20260929 18:32:00.340839 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790cfecf72db4fb686045a0fd1e05919" candidate_uuid: "14ff559b20e84d99ad466dcfdec6c361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc25d37ae32c422db047945e8b4aa443"
I20260929 18:32:00.340969 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790cfecf72db4fb686045a0fd1e05919" candidate_uuid: "14ff559b20e84d99ad466dcfdec6c361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796681cc50e74a5788883c41b9a74d06"
I20260929 18:32:00.341014 10902 raft_consensus.cc:3037] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:00.341109 10768 raft_consensus.cc:3037] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:00.342042 10902 raft_consensus.cc:2454] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ff559b20e84d99ad466dcfdec6c361 in term 1.
I20260929 18:32:00.342046 10768 raft_consensus.cc:2454] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ff559b20e84d99ad466dcfdec6c361 in term 1.
I20260929 18:32:00.342343 10971 leader_election.cc:302] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ff559b20e84d99ad466dcfdec6c361, 796681cc50e74a5788883c41b9a74d06; no voters:
I20260929 18:32:00.342504 11106 raft_consensus.cc:2781] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:00.342729 11106 raft_consensus.cc:686] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 1 LEADER]: Becoming Leader. State: Replica: 14ff559b20e84d99ad466dcfdec6c361, State: Running, Role: LEADER
I20260929 18:32:00.343055 11106 consensus_queue.cc:237] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } }
I20260929 18:32:00.346089 10622 catalog_manager.cc:5495] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ff559b20e84d99ad466dcfdec6c361 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14ff559b20e84d99ad466dcfdec6c361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ff559b20e84d99ad466dcfdec6c361" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:00.360038 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.356515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.0.98.3:34193)
0929 18:32:00.356627 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.356636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.356651 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:00.356794 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.356796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.356846 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.356930 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:00.357198 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.357680 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.358561 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.358782 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.358860 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.359149 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:00.359568 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:00.359693 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.359763 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.359808 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:32:00.365587 10768 raft_consensus.cc:1261] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 1 FOLLOWER]: Refusing update from remote peer 14ff559b20e84d99ad466dcfdec6c361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:00.365587 10902 raft_consensus.cc:1261] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 1 FOLLOWER]: Refusing update from remote peer 14ff559b20e84d99ad466dcfdec6c361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:00.366290 11109 consensus_queue.cc:1035] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796681cc50e74a5788883c41b9a74d06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:00.366473 11106 consensus_queue.cc:1035] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc25d37ae32c422db047945e8b4aa443" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:00.374709 11113 mvcc.cc:205] Tried to move back new op lower bound from 7334734726610472960 to 7334734726528053248. Current Snapshot: MvccSnapshot[applied={T|T < 7334734726610472960}]
I20260929 18:32:00.376096 11114 mvcc.cc:205] Tried to move back new op lower bound from 7334734726610472960 to 7334734726528053248. Current Snapshot: MvccSnapshot[applied={T|T < 7334734726610472960}]
I20260929 18:32:00.381783 10623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:00.381955 10623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:00.384186 10623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 588519492e2044cca663454b2c1904b7: Sending DeleteTablet for 3 replicas of tablet 790cfecf72db4fb686045a0fd1e05919
I20260929 18:32:00.384897 10882 tablet_service.cc:1514] Processing DeleteTablet for tablet 790cfecf72db4fb686045a0fd1e05919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54466
I20260929 18:32:00.385085 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet 790cfecf72db4fb686045a0fd1e05919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34650
I20260929 18:32:00.385468 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet 790cfecf72db4fb686045a0fd1e05919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42056
I20260929 18:32:00.385555 11123 tablet_replica.cc:330] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: stopping tablet replica
I20260929 18:32:00.385897 11124 tablet_replica.cc:330] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06: stopping tablet replica
I20260929 18:32:00.385900 11123 raft_consensus.cc:2229] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:00.386051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10685
I20260929 18:32:00.386260 11124 raft_consensus.cc:2229] T 790cfecf72db4fb686045a0fd1e05919 P 796681cc50e74a5788883c41b9a74d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:00.387604 11123 raft_consensus.cc:2258] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:00.388984 11123 ts_tablet_manager.cc:1905] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:00.391680 11123 ts_tablet_manager.cc:1918] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:00.391788 11123 log.cc:1196] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/790cfecf72db4fb686045a0fd1e05919
I20260929 18:32:00.392014 11122 tablet_replica.cc:330] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: stopping tablet replica
I20260929 18:32:00.392261 11123 ts_tablet_manager.cc:1939] T 790cfecf72db4fb686045a0fd1e05919 P 14ff559b20e84d99ad466dcfdec6c361: Deleting consensus metadata
I20260929 18:32:00.392392 11122 raft_consensus.cc:2229] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:00.392751 11122 raft_consensus.cc:2258] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:00.393563 10609 catalog_manager.cc:4841] TS 14ff559b20e84d99ad466dcfdec6c361 (127.0.98.3:34193): tablet 790cfecf72db4fb686045a0fd1e05919 (table test-table [id=fa432e5be506488c96d0a8e12393a4d7]) successfully deleted
W20260929 18:32:00.394186 10609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39925 (error 108)
I20260929 18:32:00.394202 11122 ts_tablet_manager.cc:1905] T 790cfecf72db4fb686045a0fd1e05919 P fc25d37ae32c422db047945e8b4aa443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:00.394320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10819
I20260929 18:32:00.396637 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.396289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.0.1:42070)
0929 18:32:00.396470 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.396536 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 18:32:00.396871 10609 catalog_manager.cc:4584] TS 796681cc50e74a5788883c41b9a74d06 (127.0.98.1:39925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 790cfecf72db4fb686045a0fd1e05919: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39925: connect: Connection refused (error 111)
I20260929 18:32:00.400488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10953
I20260929 18:32:00.401605 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.400711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43857 (local address 127.0.0.1:54478)
0929 18:32:00.401435 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.401498 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627}
I20260929 18:32:00.406453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 10591
2026-09-29T18:32:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2194 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[11128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[11128](info): set up 2 sockets
Sep 29 18:32:00 dist-test-slave-fmv8 krb5kdc[11128](info): commencing operation
krb5kdc: starting...
W20260929 18:32:01.453226 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:32:01 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 18:32:01.616976 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6eIoKd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/postgres -l logfile start
2026-09-29 18:32:03.985 UTC [11158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:32:03.986 UTC [11158] LOG: listening on IPv4 address "127.0.98.20", port 46987
2026-09-29 18:32:03.989 UTC [11158] LOG: listening on Unix socket "/tmp/.s.PGSQL.46987"
2026-09-29 18:32:04.024 UTC [11161] LOG: database system was shut down at 2026-09-29 18:32:02 UTC
2026-09-29 18:32:04.029 UTC [11158] LOG: database system is ready to accept connections
I20260929 18:32:04.976996 392 mini_postgres.cc:96] Postgres bound to 46987
2026-09-29 18:32:04.983 UTC [11169] FATAL: database "slave" does not exist
127.0.98.20:46987 - accepting connections
I20260929 18:32:04.983713 392 mini_ranger.cc:161] Starting Ranger...
I20260929 18:32:04.998569 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:32:05.460011 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:32:05.460088 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin
2026-09-29 18:32:05,766 [I] DB FLAVOR :POSTGRES
2026-09-29 18:32:05,767 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:32:05,767 [I] Checking connection..
2026-09-29 18:32:05,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:32:06,012 [I] Checking connection passed.
2026-09-29 18:32:06,012 [I] --------- Verifying version history table ---------
2026-09-29 18:32:06,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:06,247 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:32:06,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:06,478 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:32:06,479 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:32:06,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:32:06.716 UTC [11233] WARNING: there is no transaction in progress
2026-09-29 18:32:06,723 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:32:06,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:06,995 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:32:06,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:32:06,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:32:07,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:32:07,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fmv8.c.gcp-upstream.internal','N') ;"
2026-09-29 18:32:07,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:32:07,998 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:32:07,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:32:08,238 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:32:08,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:32:08,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:32:08,695 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:32:08,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:32:09.634 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.650 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.667 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.682 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.698 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.713 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.766 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.772 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.786 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.796 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.806 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.820 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.829 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.835 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.842 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:09.849 UTC [11341] WARNING: there is no transaction in progress
2026-09-29 18:32:10,477 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:32:10,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:32:10,711 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:32:10,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:32:10,961 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:32:10,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:32:11,220 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:32:11,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:32:11,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:32:11,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fmv8.c.gcp-upstream.internal';"
2026-09-29 18:32:11,914 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:32:11,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:32:12,148 [I] DB_PATCHES have already been applied
I20260929 18:32:12.154047 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6eIoKd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:32:12.154109 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 18:32:12.157413 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50557
Sep 29, 2026 6:32:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:32:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:32:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:32:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:32:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:32:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:32:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:32:13.258476 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.275s user 0.005s sys 0.005s
Sep 29, 2026 6:32:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:32:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:32:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:32:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:32:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:32:34.018818 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:32:34 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:32:34.103025 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:32:34.112298 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:32:34.923399 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:32:34.923496 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms
I20260929 18:32:34.923591 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:46987
2026-09-29 18:32:35,100 [I] DB FLAVOR :POSTGRES
2026-09-29 18:32:35,101 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:32:35,101 [I] Checking connection
2026-09-29 18:32:35,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:32:35,358 [I] connection success
2026-09-29 18:32:35,358 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:32:35,358 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:32:35,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:32:35,593 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:32:35,593 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:32:35,593 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:32:35,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:32:35,871 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:32:36.339933 392 mini_ranger_kms.cc:310] Created kms service
I20260929 18:32:36.521173 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:32:36.565017 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:32:36 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:36.972697 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:32:36.973043 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:32:36.973070 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 18:32:36.976884 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11521
I20260929 18:32:36.977005 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40419
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:32:39.216888 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.114s user 0.010s sys 0.151s
I20260929 18:32:39.217028 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:32:39.217126 392 mini_ranger_kms.cc:401] 127.0.98.20:40419/kms/v1/keys
Sep 29 18:32:39 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:32:39 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:32:40 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:32:42 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:32:42.225208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36797
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40419/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:42.354590 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:42.354828 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:42.354866 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:42.358824 11572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:42.358877 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:42.358897 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:42.358913 11572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:42.358928 11572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:42.364042 11572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40419/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36797
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:32:42.364995 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:42.365942 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:42.374622 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:42.374703 11572 server_base.cc:1034] running on GCE node
W20260929 18:32:42.374644 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:42.374678 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:42.375432 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:42.376583 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:42.377774 11572 hybrid_clock.cc:648] HybridClock initialized: now 1790706762377755 us; error 30 us; skew 500 ppm
Sep 29 18:32:42 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:42.385658 11572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:32:42.386466 11572 webserver.cc:460] Webserver started at http://127.0.98.62:39879/ using document root <none> and password file <none>
I20260929 18:32:42.386955 11572 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:42.387004 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:42.387166 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:32:42 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:42.441480 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "85bd544a3620495b93b9590ff5290eab"
format_stamp: "Formatted at 2026-09-29 18:32:42 on dist-test-slave-fmv8"
server_key: "48dc801a7e98890b1702e6561cd9492f"
server_key_iv: "edd15ffe93858d3d4acb6b5073b45eb2"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:42.442195 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "85bd544a3620495b93b9590ff5290eab"
format_stamp: "Formatted at 2026-09-29 18:32:42 on dist-test-slave-fmv8"
server_key: "48dc801a7e98890b1702e6561cd9492f"
server_key_iv: "edd15ffe93858d3d4acb6b5073b45eb2"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:42.446957 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:32:42.463361 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:42.465072 11572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:42.465207 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "85bd544a3620495b93b9590ff5290eab"
format_stamp: "Formatted at 2026-09-29 18:32:42 on dist-test-slave-fmv8"
server_key: "48dc801a7e98890b1702e6561cd9492f"
server_key_iv: "edd15ffe93858d3d4acb6b5073b45eb2"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:42.465335 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:42.483886 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:42.486948 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:42.487128 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:42.496007 11572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36797
I20260929 18:32:42.496021 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36797 every 8 connection(s)
I20260929 18:32:42.497334 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:42.500191 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:42.502928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 11572
I20260929 18:32:42.503047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:32:42.503337 392 external_mini_cluster.cc:1428] Setting key 62f6aa3054b2a3213d28cc7c36f36305
I20260929 18:32:42.506243 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: Bootstrap starting.
I20260929 18:32:42.509140 11640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:42.509841 11640 log.cc:824] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:42.511910 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: No bootstrap required, opened a new log
Sep 29 18:32:42 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:42.515818 11640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } }
I20260929 18:32:42.516073 11640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:42.516155 11640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bd544a3620495b93b9590ff5290eab, State: Initialized, Role: FOLLOWER
I20260929 18:32:42.516640 11640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } }
I20260929 18:32:42.516822 11640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:42.516914 11640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:42.517025 11640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:42.518096 11640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } }
I20260929 18:32:42.518445 11640 leader_election.cc:302] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bd544a3620495b93b9590ff5290eab; no voters:
I20260929 18:32:42.518757 11640 leader_election.cc:288] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:42.518893 11645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:42.519248 11645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [term 1 LEADER]: Becoming Leader. State: Replica: 85bd544a3620495b93b9590ff5290eab, State: Running, Role: LEADER
I20260929 18:32:42.519527 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:42.504790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.0.98.62:36797)
0929 18:32:42.505227 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:42.505245 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:42.505287 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:32:42.506121 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:42.506145 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:42.506580 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:42.506946 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:42.508259 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:42.509819 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:42.510844 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:42.511533 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:42.511695 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:42.514711 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:42.514717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:42.514740 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:42.514765 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:42.516452 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:42.516893 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:42.516899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:42.516904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:42.517038 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:42.517319 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:42.517324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:42.517325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:42.517815 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:42.518019 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:42.518263 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 18:32:42.518553 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260929 18:32:42.519549 11645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } }
I20260929 18:32:42.519990 11640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:42.520856 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bd544a3620495b93b9590ff5290eab. Latest consensus state: current_term: 1 leader_uuid: "85bd544a3620495b93b9590ff5290eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } } }
I20260929 18:32:42.521067 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:42.521252 11646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bd544a3620495b93b9590ff5290eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bd544a3620495b93b9590ff5290eab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36797 } } }
I20260929 18:32:42.521553 11646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:42.522058 11640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:32:42.522820 11654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:42.528714 11654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:42.535656 11654 catalog_manager.cc:1347] Generated new cluster ID: ef3235ca0dbe4b53ae863dd3791d84c2
I20260929 18:32:42.535714 11654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:42.574424 11654 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:42.575387 11654 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:42.586269 11654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: Generated new TSK 0
I20260929 18:32:42.586989 11654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:32:43.235423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:43.389997 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:43.390269 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:43.390343 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:43.394379 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:43.394459 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:43.394548 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:32:43.400801 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:32:43.402074 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:43.403371 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:43.415369 11695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:43.415491 11692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:43.415697 11693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:43.416648 11685 server_base.cc:1034] running on GCE node
I20260929 18:32:43.417088 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:43.417783 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:43.418967 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790706763418892 us; error 92 us; skew 500 ppm
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:43.422542 11685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:32:43.423676 11685 webserver.cc:460] Webserver started at http://127.0.98.1:45677/ using document root <none> and password file <none>
I20260929 18:32:43.424417 11685 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:43.424505 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:43.424790 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:43.441501 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "cafa136be3e846c2b33ac94a53f48724"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "034d54db1271f061d86a881d18008185"
server_key_iv: "65f8ec36d79b86c1a02f313a15359a71"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.442432 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cafa136be3e846c2b33ac94a53f48724"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "034d54db1271f061d86a881d18008185"
server_key_iv: "65f8ec36d79b86c1a02f313a15359a71"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.449627 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 18:32:43.468575 11702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.470227 11685 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 18:32:43.470366 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "cafa136be3e846c2b33ac94a53f48724"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "034d54db1271f061d86a881d18008185"
server_key_iv: "65f8ec36d79b86c1a02f313a15359a71"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.470511 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:43.503549 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:43.509114 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:43.509352 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:43.510028 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:43.511068 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:43.511176 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.511297 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:43.511376 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.521651 11685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41365
I20260929 18:32:43.522817 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:43.525195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 11685
I20260929 18:32:43.525305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:43.525626 392 external_mini_cluster.cc:1428] Setting key 29677ef1385bda4bf240a237322aabaf
I20260929 18:32:43.530326 11817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41365 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:43.551767 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:43.533098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46965 (local address 127.0.98.62:36797)
0929 18:32:43.533410 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:43.533419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:43.537627 (+ 4208us) server_negotiation.cc:407] Connection header received
0929 18:32:43.539002 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:43.539008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:43.539075 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:43.539143 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:43.540753 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.541453 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.543185 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.543618 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.543715 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:43.546988 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:43.546993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:43.546996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:43.547029 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:43.549117 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:43.549777 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:43.549783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:43.549785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:43.549882 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:43.550308 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:43.550313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:43.550315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:43.550548 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:43.550637 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:43.551364 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 18:32:43.551534 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260929 18:32:43.553293 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:43.533343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36797 (local address 127.0.98.1:46965)
0929 18:32:43.537451 (+ 4108us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:43.537515 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:43.538716 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:43.539601 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:43.539610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:43.539973 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:43.540602 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:43.540617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.541594 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.541598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:43.543043 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:43.543056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.543970 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.543976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:43.544239 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:43.544929 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:43.544954 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:43.546769 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:43.549261 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:43.549271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:43.549288 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:43.549654 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:43.550008 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:43.550012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:43.550014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:43.550190 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:43.550709 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:43.550717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:43.551057 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 18:32:43.551997 (+ 940us) client_negotiation.cc:240] Negotiation successful
0929 18:32:43.552352 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3842,"thread_start_us":147,"threads_started":1}
I20260929 18:32:43.554843 11819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36797
I20260929 18:32:43.555116 11819 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:43.555655 11819 heartbeater.cc:511] Master 127.0.98.62:36797 requested a full tablet report, sending...
I20260929 18:32:43.557277 11604 ts_manager.cc:194] Registered new tserver with Master: cafa136be3e846c2b33ac94a53f48724 (127.0.98.1:41365)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:32:43.558743 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46965
I20260929 18:32:43.559376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:32:43.693894 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:43.694175 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:43.694303 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:43.698508 11824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:43.698585 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:43.698683 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:32:43.704097 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:32:43.705199 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:43.706280 11824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:43.715910 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:43.715911 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:43.716431 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:43.716743 11824 server_base.cc:1034] running on GCE node
I20260929 18:32:43.717298 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:43.717999 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:43.719784 11824 hybrid_clock.cc:648] HybridClock initialized: now 1790706763719721 us; error 80 us; skew 500 ppm
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:43.723105 11824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:32:43.724184 11824 webserver.cc:460] Webserver started at http://127.0.98.2:41691/ using document root <none> and password file <none>
I20260929 18:32:43.724747 11824 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:43.724839 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:43.725070 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:43.741636 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d3648f6eaee4a7ab80a5eb306678603"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "5226ad4a13442ec4b26bb3aaf56ea4a9"
server_key_iv: "4798ca115b7cffd20435631ac9d3d05c"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.742444 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d3648f6eaee4a7ab80a5eb306678603"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "5226ad4a13442ec4b26bb3aaf56ea4a9"
server_key_iv: "4798ca115b7cffd20435631ac9d3d05c"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.749691 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 18:32:43.779062 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.780824 11824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:43.781041 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "4d3648f6eaee4a7ab80a5eb306678603"
format_stamp: "Formatted at 2026-09-29 18:32:43 on dist-test-slave-fmv8"
server_key: "5226ad4a13442ec4b26bb3aaf56ea4a9"
server_key_iv: "4798ca115b7cffd20435631ac9d3d05c"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:43.781278 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:43.795969 11824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:43.799842 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:43.800120 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:43.800877 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:43.802111 11824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:43.802222 11824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.802333 11824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:43.802402 11824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:43.819380 11824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46345
I20260929 18:32:43.820806 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:43.822407 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46345 every 8 connection(s)
I20260929 18:32:43.827700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 11824
I20260929 18:32:43.827840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:43.828200 392 external_mini_cluster.cc:1428] Setting key 780c8760396e04ee98419980df448e83
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 29 18:32:43 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:43.858664 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:43.824732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54067 (local address 127.0.98.62:36797)
0929 18:32:43.824997 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:43.825003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:43.841691 (+ 16688us) server_negotiation.cc:407] Connection header received
0929 18:32:43.843212 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:43.843217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:43.843279 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:43.843340 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:43.845590 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.846539 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.848501 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.848908 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.848999 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:43.854307 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:43.854311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:43.854314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:43.854347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:43.856217 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:43.856850 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:43.856854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:43.856856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:43.856939 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:43.857319 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:43.857323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:43.857325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:43.857558 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:43.857640 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:43.858342 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 18:32:43.858490 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 18:32:43.860631 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:43.825273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36797 (local address 127.0.98.2:54067)
0929 18:32:43.841447 (+ 16174us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:43.841544 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:43.842905 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:43.843836 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:43.843872 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:43.844603 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:43.845386 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:43.845411 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.846718 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.846722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:43.848330 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:43.848342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:43.849234 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:43.849239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:43.849483 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:43.852268 (+ 2785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:43.852292 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:43.854088 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:43.856358 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:43.856365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:43.856392 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:43.856728 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:43.857044 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:43.857047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:43.857049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:43.857215 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:43.857707 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:43.857713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:43.858062 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 18:32:43.859134 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0929 18:32:43.859497 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15801,"thread_start_us":149,"threads_started":1}
I20260929 18:32:43.862310 11954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36797
I20260929 18:32:43.862571 11954 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:43.863126 11954 heartbeater.cc:511] Master 127.0.98.62:36797 requested a full tablet report, sending...
I20260929 18:32:43.864288 11604 ts_manager.cc:194] Registered new tserver with Master: 4d3648f6eaee4a7ab80a5eb306678603 (127.0.98.2:46345)
I20260929 18:32:43.864986 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:32:43.875517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:44.053349 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:44.053917 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:44.054031 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:44.060122 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:44.060214 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:44.060312 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:32:44.068567 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:32:44.070011 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:44.071410 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:44.084602 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.085086 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.085307 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.086550 11959 server_base.cc:1034] running on GCE node
I20260929 18:32:44.087100 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:44.087893 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:44.089094 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790706764089027 us; error 84 us; skew 500 ppm
Sep 29 18:32:44 dist-test-slave-fmv8 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:44.092648 11959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:32:44.093755 11959 webserver.cc:460] Webserver started at http://127.0.98.3:43303/ using document root <none> and password file <none>
I20260929 18:32:44.094482 11959 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:44.094578 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:44.094830 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:32:44 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:44.116792 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "9342ead49672428581a57d7101562aa7"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-fmv8"
server_key: "ee7c50afdebe7285e693cab1ba27ed68"
server_key_iv: "601c47f03fa2b43d64f21c5367003f5a"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:44.117491 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9342ead49672428581a57d7101562aa7"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-fmv8"
server_key: "ee7c50afdebe7285e693cab1ba27ed68"
server_key_iv: "601c47f03fa2b43d64f21c5367003f5a"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:44.128430 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.001s
I20260929 18:32:44.148074 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.149837 11959 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:32:44.149988 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "9342ead49672428581a57d7101562aa7"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-fmv8"
server_key: "ee7c50afdebe7285e693cab1ba27ed68"
server_key_iv: "601c47f03fa2b43d64f21c5367003f5a"
server_key_version: "kuduclusterkey@0"
I20260929 18:32:44.150141 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 18:32:44 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:32:44.206488 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:44.213490 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:44.213686 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:44.214411 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:44.215446 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:44.215512 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.215585 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:44.215617 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.245698 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39161
I20260929 18:32:44.246831 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:44.249339 12087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39161 every 8 connection(s)
I20260929 18:32:44.256955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 11959
I20260929 18:32:44.257081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:44.257359 392 external_mini_cluster.cc:1428] Setting key c4567a85f49458afccb9e09b900dc742
Sep 29 18:32:44 dist-test-slave-fmv8 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:32:44.271054 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.251302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38225 (local address 127.0.98.62:36797)
0929 18:32:44.251453 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.251458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.256216 (+ 4758us) server_negotiation.cc:407] Connection header received
0929 18:32:44.257683 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.257689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.257750 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.257808 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:44.259450 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.260318 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.262042 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.262396 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.262478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.265523 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:44.265528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:44.265531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:44.265557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:44.267333 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.267960 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.267965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.267967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.268050 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.268513 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.268518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.268519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.268745 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:44.268822 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.270736 (+ 1914us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.270870 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:32:44.272702 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.251642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36797 (local address 127.0.98.3:38225)
0929 18:32:44.255994 (+ 4352us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.256057 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.257418 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.258239 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.258249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.258610 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:44.259281 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.259304 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.260468 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.260472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.261895 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.261905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.262678 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.262682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.262887 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.263590 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:44.263610 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:44.265319 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:44.267473 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.267479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.267495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.267820 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.268194 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.268198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.268202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.268392 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.268904 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:44.268910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:44.269232 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.271292 (+ 2060us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.271624 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4083,"thread_start_us":128,"threads_started":1}
I20260929 18:32:44.273975 12088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36797
I20260929 18:32:44.274230 12088 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:44.274787 12088 heartbeater.cc:511] Master 127.0.98.62:36797 requested a full tablet report, sending...
I20260929 18:32:44.275863 11604 ts_manager.cc:194] Registered new tserver with Master: 9342ead49672428581a57d7101562aa7 (127.0.98.3:39161)
I20260929 18:32:44.276630 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38225
I20260929 18:32:44.283119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:44.295639 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.286185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.62:36797)
0929 18:32:44.286297 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.286301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.286911 (+ 610us) server_negotiation.cc:407] Connection header received
0929 18:32:44.287070 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.287073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.287121 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.287202 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:44.288223 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.288978 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.290117 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.290420 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.290521 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.292180 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:44.292184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:44.292187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:44.292214 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:44.293886 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.294361 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.294364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.294366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.294461 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.294744 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.294748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.294749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.294981 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:44.295116 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.295368 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.295485 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:32:44.299052 11604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:44.303500 11604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:44.322782 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.314615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39161 (local address 127.0.0.1:58956)
0929 18:32:44.316355 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.316406 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.316535 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.317529 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.317535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.317558 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:44.317686 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.317693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.319470 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.319473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.320895 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.320905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.322226 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.322230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.322356 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.322395 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.322504 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.322619 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"thread_start_us":95,"threads_started":1}
I20260929 18:32:44.323982 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.314796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.0.98.3:39161)
0929 18:32:44.316848 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.316860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.316885 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:32:44.316969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.316974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.317185 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.317353 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:44.318088 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.319328 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.321060 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.321911 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.322015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.322165 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.323716 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.323843 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":145,"threads_started":1}
I20260929 18:32:44.325353 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.315333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41365 (local address 127.0.0.1:57304)
0929 18:32:44.316422 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.316438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.317768 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.318485 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.318489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.318508 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:44.318617 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.318624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.321293 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.321296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.322478 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.322489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.324955 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.324962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.325060 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.325084 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.325157 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.325224 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":116,"threads_started":1}
I20260929 18:32:44.325850 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.315797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.98.1:41365)
0929 18:32:44.316254 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.316263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.317877 (+ 1614us) server_negotiation.cc:407] Connection header received
0929 18:32:44.317964 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.317970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.318176 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.318319 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:44.318757 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.321173 (+ 2416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.322623 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.323712 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.323837 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.323973 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.325610 (+ 1637us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.325710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":123,"threads_started":1}
I20260929 18:32:44.327170 12016 tablet_service.cc:1467] Processing CreateTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 (DEFAULT_TABLE table=test-table [id=df558f20c65247ffa6ee68e2e1f94ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:44.328368 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e70efd9c6a40aab50486f34c98f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:44.329039 11750 tablet_service.cc:1467] Processing CreateTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 (DEFAULT_TABLE table=test-table [id=df558f20c65247ffa6ee68e2e1f94ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:44.330027 11750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e70efd9c6a40aab50486f34c98f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:44.339218 12107 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: Bootstrap starting.
I20260929 18:32:44.339639 12106 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Bootstrap starting.
I20260929 18:32:44.341550 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.315391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46345 (local address 127.0.0.1:33808)
0929 18:32:44.317891 (+ 2500us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.317904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.317977 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.336831 (+ 18854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.336835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.336859 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:44.336969 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.336976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.338705 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.338709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.339951 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.339962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.341153 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.341157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.341251 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.341280 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.341339 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.341409 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2441,"thread_start_us":95,"threads_started":1}
I20260929 18:32:44.342217 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.315807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.0.98.2:46345)
0929 18:32:44.323266 (+ 7459us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.323280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.323303 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:32:44.323383 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.323388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.323564 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.337137 (+ 13573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:44.337293 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.338488 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.340126 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.340962 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.341815 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.341958 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.342004 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.342088 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7350,"thread_start_us":101,"threads_started":1}
I20260929 18:32:44.345113 11887 tablet_service.cc:1467] Processing CreateTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 (DEFAULT_TABLE table=test-table [id=df558f20c65247ffa6ee68e2e1f94ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:44.346153 11887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e70efd9c6a40aab50486f34c98f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:44.346185 12107 tablet_bootstrap.cc:654] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:44.347165 12107 log.cc:824] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:44.347306 12106 tablet_bootstrap.cc:654] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:44.348119 12106 log.cc:824] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:44.351331 12107 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: No bootstrap required, opened a new log
I20260929 18:32:44.351586 12107 ts_tablet_manager.cc:1397] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260929 18:32:44.357095 12107 raft_consensus.cc:348] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.357560 12107 raft_consensus.cc:374] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:44.357661 12107 raft_consensus.cc:729] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafa136be3e846c2b33ac94a53f48724, State: Initialized, Role: FOLLOWER
I20260929 18:32:44.358233 12107 consensus_queue.cc:260] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.359411 12107 ts_tablet_manager.cc:1428] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260929 18:32:44.360859 12106 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: No bootstrap required, opened a new log
I20260929 18:32:44.361104 12106 ts_tablet_manager.cc:1397] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20260929 18:32:44.365988 12106 raft_consensus.cc:348] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.366324 12106 raft_consensus.cc:374] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:44.366431 12106 raft_consensus.cc:729] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9342ead49672428581a57d7101562aa7, State: Initialized, Role: FOLLOWER
I20260929 18:32:44.367003 12106 consensus_queue.cc:260] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.368080 12106 ts_tablet_manager.cc:1428] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260929 18:32:44.368706 12088 heartbeater.cc:503] Master 127.0.98.62:36797 was elected leader, sending a full tablet report...
I20260929 18:32:44.372442 11819 heartbeater.cc:503] Master 127.0.98.62:36797 was elected leader, sending a full tablet report...
I20260929 18:32:44.378894 12110 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Bootstrap starting.
I20260929 18:32:44.381376 12110 tablet_bootstrap.cc:654] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:44.382774 12110 log.cc:824] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:44.385429 12110 tablet_bootstrap.cc:492] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: No bootstrap required, opened a new log
I20260929 18:32:44.385612 12110 ts_tablet_manager.cc:1397] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:32:44.389919 12110 raft_consensus.cc:348] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.390271 12110 raft_consensus.cc:374] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:44.390367 12110 raft_consensus.cc:729] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3648f6eaee4a7ab80a5eb306678603, State: Initialized, Role: FOLLOWER
I20260929 18:32:44.390913 12110 consensus_queue.cc:260] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.393024 12110 ts_tablet_manager.cc:1428] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:32:44.393445 11954 heartbeater.cc:503] Master 127.0.98.62:36797 was elected leader, sending a full tablet report...
I20260929 18:32:44.427232 12112 raft_consensus.cc:482] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:44.427493 12112 raft_consensus.cc:504] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.428898 12112 leader_election.cc:288] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3648f6eaee4a7ab80a5eb306678603 (127.0.98.2:46345), cafa136be3e846c2b33ac94a53f48724 (127.0.98.1:41365)
I20260929 18:32:44.435118 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.429295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46345 (local address 127.0.98.3:47837)
0929 18:32:44.430515 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.430531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.430628 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.431340 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.431343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.431361 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:44.431478 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.431485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.432489 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.432492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.433745 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.433755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.434722 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.434725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.434826 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.434849 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.434918 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.434981 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102}
I20260929 18:32:44.435559 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.430872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47837 (local address 127.0.98.2:46345)
0929 18:32:44.431009 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.431013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.431030 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:44.431082 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.431085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.431134 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.431215 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:44.431608 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.432350 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.433902 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.434506 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.434604 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.434664 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.435374 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.435449 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:32:44.436323 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e70efd9c6a40aab50486f34c98f3c1" candidate_uuid: "9342ead49672428581a57d7101562aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3648f6eaee4a7ab80a5eb306678603" is_pre_election: true
I20260929 18:32:44.436638 11907 raft_consensus.cc:2454] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9342ead49672428581a57d7101562aa7 in term 0.
I20260929 18:32:44.437130 11977 leader_election.cc:302] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3648f6eaee4a7ab80a5eb306678603, 9342ead49672428581a57d7101562aa7; no voters:
I20260929 18:32:44.437420 12112 raft_consensus.cc:2781] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:44.437497 12112 raft_consensus.cc:482] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:44.437539 12112 raft_consensus.cc:3037] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:44.437606 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.431016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53081 (local address 127.0.98.1:41365)
0929 18:32:44.431128 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.431132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.431564 (+ 432us) server_negotiation.cc:407] Connection header received
0929 18:32:44.431737 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.431740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.431787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.431830 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:44.432234 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.432939 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.434464 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.435103 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.435180 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.435236 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.437399 (+ 2163us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.437492 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:32:44.438192 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.429757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41365 (local address 127.0.98.3:53081)
0929 18:32:44.431478 (+ 1721us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.431492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.431626 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.431947 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.431950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.431964 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:44.432073 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.432079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.433084 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.433087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.434313 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.434323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.437079 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.437083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.437181 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.437204 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.437935 (+ 731us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.438013 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"thread_start_us":401,"threads_started":1}
I20260929 18:32:44.438864 12112 raft_consensus.cc:504] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.438889 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e70efd9c6a40aab50486f34c98f3c1" candidate_uuid: "9342ead49672428581a57d7101562aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cafa136be3e846c2b33ac94a53f48724" is_pre_election: true
I20260929 18:32:44.439158 11753 raft_consensus.cc:2454] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9342ead49672428581a57d7101562aa7 in term 0.
I20260929 18:32:44.439203 12112 leader_election.cc:288] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3648f6eaee4a7ab80a5eb306678603 (127.0.98.2:46345), cafa136be3e846c2b33ac94a53f48724 (127.0.98.1:41365)
I20260929 18:32:44.439661 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e70efd9c6a40aab50486f34c98f3c1" candidate_uuid: "9342ead49672428581a57d7101562aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cafa136be3e846c2b33ac94a53f48724"
I20260929 18:32:44.439771 11753 raft_consensus.cc:3037] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:44.440127 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e70efd9c6a40aab50486f34c98f3c1" candidate_uuid: "9342ead49672428581a57d7101562aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3648f6eaee4a7ab80a5eb306678603"
I20260929 18:32:44.440267 11907 raft_consensus.cc:3037] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:44.441059 11753 raft_consensus.cc:2454] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9342ead49672428581a57d7101562aa7 in term 1.
I20260929 18:32:44.441473 11907 raft_consensus.cc:2454] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9342ead49672428581a57d7101562aa7 in term 1.
I20260929 18:32:44.441787 11976 leader_election.cc:302] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9342ead49672428581a57d7101562aa7, cafa136be3e846c2b33ac94a53f48724; no voters:
I20260929 18:32:44.442073 12112 raft_consensus.cc:2781] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:44.442291 12112 raft_consensus.cc:686] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 1 LEADER]: Becoming Leader. State: Replica: 9342ead49672428581a57d7101562aa7, State: Running, Role: LEADER
I20260929 18:32:44.442624 12112 consensus_queue.cc:237] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } }
I20260929 18:32:44.445919 11604 catalog_manager.cc:5495] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9342ead49672428581a57d7101562aa7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9342ead49672428581a57d7101562aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9342ead49672428581a57d7101562aa7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:44.476936 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.471608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.0.98.3:39161)
0929 18:32:44.471762 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.471767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.471873 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:32:44.472046 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.472049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.472120 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.472252 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:44.472513 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.473283 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.474697 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.475045 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.475143 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.475579 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:44.476177 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:44.476308 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.476734 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.476795 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:32:44.483943 11753 raft_consensus.cc:1261] T b5e70efd9c6a40aab50486f34c98f3c1 P cafa136be3e846c2b33ac94a53f48724 [term 1 FOLLOWER]: Refusing update from remote peer 9342ead49672428581a57d7101562aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:44.484746 12116 consensus_queue.cc:1035] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cafa136be3e846c2b33ac94a53f48724" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:44.486059 11907 raft_consensus.cc:1261] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 1 FOLLOWER]: Refusing update from remote peer 9342ead49672428581a57d7101562aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:44.490468 12116 consensus_queue.cc:1035] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3648f6eaee4a7ab80a5eb306678603" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:44.506534 12120 mvcc.cc:205] Tried to move back new op lower bound from 7334734907317551104 to 7334734907159912448. Current Snapshot: MvccSnapshot[applied={T|T < 7334734907317551104}]
I20260929 18:32:44.509888 11604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:44.510046 11604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:44.512620 11604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85bd544a3620495b93b9590ff5290eab: Sending DeleteTablet for 3 replicas of tablet b5e70efd9c6a40aab50486f34c98f3c1
I20260929 18:32:44.513461 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58956
I20260929 18:32:44.513496 11750 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57304
I20260929 18:32:44.513583 11887 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e70efd9c6a40aab50486f34c98f3c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33808
I20260929 18:32:44.514590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 11685
I20260929 18:32:44.518862 12128 tablet_replica.cc:330] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: stopping tablet replica
I20260929 18:32:44.519166 12128 raft_consensus.cc:2229] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:44.519497 12128 raft_consensus.cc:2258] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:44.519909 12130 tablet_replica.cc:330] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: stopping tablet replica
I20260929 18:32:44.520221 12130 raft_consensus.cc:2229] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:44.520534 12130 raft_consensus.cc:2258] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:44.521553 12128 ts_tablet_manager.cc:1905] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:44.524418 12128 ts_tablet_manager.cc:1918] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:44.524581 12128 log.cc:1196] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/b5e70efd9c6a40aab50486f34c98f3c1
I20260929 18:32:44.524998 12128 ts_tablet_manager.cc:1939] T b5e70efd9c6a40aab50486f34c98f3c1 P 9342ead49672428581a57d7101562aa7: Deleting consensus metadata
I20260929 18:32:44.525314 12130 ts_tablet_manager.cc:1905] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:44.526369 11590 catalog_manager.cc:4841] TS 9342ead49672428581a57d7101562aa7 (127.0.98.3:39161): tablet b5e70efd9c6a40aab50486f34c98f3c1 (table test-table [id=df558f20c65247ffa6ee68e2e1f94ea0]) successfully deleted
I20260929 18:32:44.527828 12130 ts_tablet_manager.cc:1918] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:44.527942 12130 log.cc:1196] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/b5e70efd9c6a40aab50486f34c98f3c1
I20260929 18:32:44.528282 12130 ts_tablet_manager.cc:1939] T b5e70efd9c6a40aab50486f34c98f3c1 P 4d3648f6eaee4a7ab80a5eb306678603: Deleting consensus metadata
I20260929 18:32:44.529453 11590 catalog_manager.cc:4841] TS 4d3648f6eaee4a7ab80a5eb306678603 (127.0.98.2:46345): tablet b5e70efd9c6a40aab50486f34c98f3c1 (table test-table [id=df558f20c65247ffa6ee68e2e1f94ea0]) successfully deleted
W20260929 18:32:44.530457 11589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41365 (error 108)
I20260929 18:32:44.534058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 11824
I20260929 18:32:44.543900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 11959
I20260929 18:32:44.544328 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.543894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41365 (local address 127.0.0.1:57312)
0929 18:32:44.544108 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.544212 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260929 18:32:44.544761 11589 catalog_manager.cc:4584] TS cafa136be3e846c2b33ac94a53f48724 (127.0.98.1:41365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5e70efd9c6a40aab50486f34c98f3c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41365: connect: Connection refused (error 111)
I20260929 18:32:44.551045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 11572
I20260929 18:32:44.557353 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:32:44.891665 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:32:44.891793 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:32:45.236613 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:32:45.236 UTC [11158] LOG: received smart shutdown request
2026-09-29 18:32:45.240 UTC [11158] LOG: background worker "logical replication launcher" (PID 11167) exited with exit code 1
2026-09-29 18:32:45.240 UTC [11162] LOG: shutting down
2026-09-29 18:32:45.395 UTC [11158] LOG: database system is shut down
2026-09-29T18:32:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45099 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:45 dist-test-slave-fmv8 krb5kdc[12141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:45 dist-test-slave-fmv8 krb5kdc[12141](info): set up 2 sockets
Sep 29 18:32:45 dist-test-slave-fmv8 krb5kdc[12141](info): commencing operation
krb5kdc: starting...
W20260929 18:32:46.556104 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:32:46 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 18:32:46.714530 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6eIoKd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/postgres -l logfile start
2026-09-29 18:32:49.054 UTC [12171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:32:49.055 UTC [12171] LOG: listening on IPv4 address "127.0.98.20", port 45529
2026-09-29 18:32:49.058 UTC [12171] LOG: listening on Unix socket "/tmp/.s.PGSQL.45529"
2026-09-29 18:32:49.079 UTC [12174] LOG: database system was shut down at 2026-09-29 18:32:47 UTC
2026-09-29 18:32:49.083 UTC [12171] LOG: database system is ready to accept connections
I20260929 18:32:50.044130 392 mini_postgres.cc:96] Postgres bound to 45529
2026-09-29 18:32:50.050 UTC [12182] FATAL: database "slave" does not exist
127.0.98.20:45529 - accepting connections
I20260929 18:32:50.051110 392 mini_ranger.cc:161] Starting Ranger...
I20260929 18:32:50.066000 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:32:50.518759 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:32:50.518841 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin
2026-09-29 18:32:50,803 [I] DB FLAVOR :POSTGRES
2026-09-29 18:32:50,803 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:32:50,803 [I] Checking connection..
2026-09-29 18:32:50,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:32:51,038 [I] Checking connection passed.
2026-09-29 18:32:51,038 [I] --------- Verifying version history table ---------
2026-09-29 18:32:51,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:51,276 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:32:51,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:51,506 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:32:51,507 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:32:51,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:32:51.740 UTC [12246] WARNING: there is no transaction in progress
2026-09-29 18:32:51,746 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:32:51,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:32:51,983 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:32:51,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:32:51,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:32:52,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:32:52,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fmv8.c.gcp-upstream.internal','N') ;"
2026-09-29 18:32:52,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:32:52,944 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:32:52,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:32:53,178 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:32:53,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:32:53,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:32:53,654 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:32:53,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:32:54.577 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.593 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.610 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.626 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.643 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.663 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.721 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.727 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.740 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.750 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.760 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.775 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.783 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.789 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.795 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:54.802 UTC [12354] WARNING: there is no transaction in progress
2026-09-29 18:32:55,399 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:32:55,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:32:55,634 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:32:55,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:32:55,883 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:32:55,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:32:56,141 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:32:56,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:32:56,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:32:56,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fmv8.c.gcp-upstream.internal';"
2026-09-29 18:32:56,861 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:32:56,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:32:57,116 [I] DB_PATCHES have already been applied
I20260929 18:32:57.122473 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6eIoKd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:32:57.122546 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 18:32:57.126068 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37257
Sep 29, 2026 6:32:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:32:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:32:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:32:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:32:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:32:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:32:58.225545 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.174s user 0.000s sys 0.009s
Sep 29, 2026 6:33:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:33:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:33:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:33:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:33:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:33:19.915001 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:33:19 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706799, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:33:19.997911 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:33:20.007203 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:33:20.817049 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:33:20.817147 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms
I20260929 18:33:20.817175 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45529
2026-09-29 18:33:20,987 [I] DB FLAVOR :POSTGRES
2026-09-29 18:33:20,987 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:33:20,987 [I] Checking connection
2026-09-29 18:33:20,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:33:21,224 [I] connection success
2026-09-29 18:33:21,224 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:33:21,225 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:33:21,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:33:21,477 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:33:21,478 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:33:21,478 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:33:21,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:33:21,738 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:33:22.154930 392 mini_ranger_kms.cc:310] Created kms service
I20260929 18:33:22.315831 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:33:22.358733 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:33:22 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706799, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:22.766103 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:33:22.766249 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6eIoKd/build/debug/bin/postgresql.jar:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6eIoKd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:33:22.766268 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 18:33:22.770388 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12534
I20260929 18:33:22.770478 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:57099
Sep 29, 2026 6:33:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:33:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:33:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:33:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:33:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:33:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 18:33:24 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706804, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:24.992501 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.995s user 0.000s sys 0.154s
I20260929 18:33:24.992645 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:33:24.992744 392 mini_ranger_kms.cc:401] 127.0.98.20:57099/kms/v1/keys
Sep 29 18:33:25 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706804, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:33:25 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706804, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:33:27 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:33:27.999603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42549
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:28.118244 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:28.118484 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:28.118521 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:28.122383 12585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:28.122434 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:28.122453 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:28.122470 12585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:28.122485 12585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:28.127759 12585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:57099/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42549
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:28.128695 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:28.129669 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:28.137928 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:28.138028 12585 server_base.cc:1034] running on GCE node
W20260929 18:33:28.137952 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:28.137928 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:28.138662 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:28.139708 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:28.140904 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790706808140893 us; error 26 us; skew 500 ppm
Sep 29 18:33:28 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:28.143496 12585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:28.144410 12585 webserver.cc:460] Webserver started at http://127.0.98.62:40421/ using document root <none> and password file <none>
I20260929 18:33:28.144951 12585 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:28.145030 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:28.145244 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:28 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:28.199872 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "0723843535114876bbc7c157e195f5d9"
format_stamp: "Formatted at 2026-09-29 18:33:28 on dist-test-slave-fmv8"
server_key: "4f66157c02ddfc7d97f6a8114eab8f5c"
server_key_iv: "5d0b916bbe8e527a576d80f7b98441da"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:28.200639 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "0723843535114876bbc7c157e195f5d9"
format_stamp: "Formatted at 2026-09-29 18:33:28 on dist-test-slave-fmv8"
server_key: "4f66157c02ddfc7d97f6a8114eab8f5c"
server_key_iv: "5d0b916bbe8e527a576d80f7b98441da"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:28.205534 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 18:33:28.219220 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:28.220640 12585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:28.220788 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0723843535114876bbc7c157e195f5d9"
format_stamp: "Formatted at 2026-09-29 18:33:28 on dist-test-slave-fmv8"
server_key: "4f66157c02ddfc7d97f6a8114eab8f5c"
server_key_iv: "5d0b916bbe8e527a576d80f7b98441da"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:28.220912 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:28.232127 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:28.234941 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:28.235132 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:28.242674 12585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42549
I20260929 18:33:28.242733 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42549 every 8 connection(s)
I20260929 18:33:28.243656 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:28.247001 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:28.247390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 12585
I20260929 18:33:28.247519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:33:28.247812 392 external_mini_cluster.cc:1428] Setting key 654c3f5628f7d657bddc823b6481a576
I20260929 18:33:28.254683 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Bootstrap starting.
Sep 29 18:33:28 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:28.257373 12653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:28.258210 12653 log.cc:824] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:28.260443 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: No bootstrap required, opened a new log
I20260929 18:33:28.263000 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:28.249295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.62:42549)
0929 18:33:28.249688 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:28.249705 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:28.249747 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:33:28.250582 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:28.250598 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:28.250905 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:28.251967 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:28.252331 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:28.253278 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:28.254117 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:28.254584 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:28.254747 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:28.257377 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:28.257382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:28.257394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:28.257418 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:28.259954 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:28.260491 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:28.260497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:28.260503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:28.260626 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:28.260918 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:28.260921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:28.260922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:28.261382 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:28.261573 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:28.261829 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 18:33:28.262099 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260929 18:33:28.264264 12653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:28.264503 12653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:28.264572 12653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0723843535114876bbc7c157e195f5d9, State: Initialized, Role: FOLLOWER
I20260929 18:33:28.265017 12653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:28.265204 12653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:28.265290 12653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:28.265405 12653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:28.266250 12653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:28.266580 12653 leader_election.cc:302] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0723843535114876bbc7c157e195f5d9; no voters:
I20260929 18:33:28.266891 12653 leader_election.cc:288] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:28.266958 12658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:28.267215 12658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0723843535114876bbc7c157e195f5d9, State: Running, Role: LEADER
I20260929 18:33:28.267498 12658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:28.268200 12653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:28.268647 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0723843535114876bbc7c157e195f5d9. Latest consensus state: current_term: 1 leader_uuid: "0723843535114876bbc7c157e195f5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260929 18:33:28.268828 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:28.269119 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0723843535114876bbc7c157e195f5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260929 18:33:28.269277 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:28.269575 12664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:28.272735 12664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:28.273109 12653 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:33:28.283089 12664 catalog_manager.cc:1347] Generated new cluster ID: dea9d6bc3c75449ea1f8c48ea2d31f52
I20260929 18:33:28.283159 12664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:28.294485 12664 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:28.295758 12664 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:28.309321 12664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Generated new TSK 0
I20260929 18:33:28.310259 12664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:33:28.991132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:29.137593 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:29.137868 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:29.137970 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:29.142042 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:29.142113 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:29.142203 12698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:29.148283 12698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:29.149324 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:29.150599 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:29.162520 12705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:29.162556 12708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:29.162896 12706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:29.163069 12698 server_base.cc:1034] running on GCE node
I20260929 18:33:29.163480 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:29.164258 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:29.165442 12698 hybrid_clock.cc:648] HybridClock initialized: now 1790706809165416 us; error 41 us; skew 500 ppm
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:29.168990 12698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:29.170212 12698 webserver.cc:460] Webserver started at http://127.0.98.1:32875/ using document root <none> and password file <none>
I20260929 18:33:29.170889 12698 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:29.170976 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:29.171267 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:29.189419 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "13df9e05338744ebbabe9601ad09bbf8"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "6a116610ed482547c1ca71cbee13e456"
server_key_iv: "8f7ec1f27040abf26928f42ccaf175e1"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.190151 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13df9e05338744ebbabe9601ad09bbf8"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "6a116610ed482547c1ca71cbee13e456"
server_key_iv: "8f7ec1f27040abf26928f42ccaf175e1"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.196559 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 18:33:29.213014 12715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.214720 12698 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:29.214844 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "13df9e05338744ebbabe9601ad09bbf8"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "6a116610ed482547c1ca71cbee13e456"
server_key_iv: "8f7ec1f27040abf26928f42ccaf175e1"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.214963 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:29.242712 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:29.246227 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:29.246464 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:29.247099 12698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:29.248112 12698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:29.248242 12698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.248365 12698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:29.248440 12698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.261056 12698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38799
I20260929 18:33:29.262229 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:29.263785 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38799 every 8 connection(s)
I20260929 18:33:29.270490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 12698
I20260929 18:33:29.270618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:29.270923 392 external_mini_cluster.cc:1428] Setting key 403b4c3ac7620f6debe05be1c439ce7c
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:33:29.289515 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.265885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40265 (local address 127.0.98.62:42549)
0929 18:33:29.268546 (+ 2661us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:29.268556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:29.270325 (+ 1769us) server_negotiation.cc:407] Connection header received
0929 18:33:29.270382 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:29.270385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:29.270443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:29.270533 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:29.277890 (+ 7357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.278786 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.280577 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.280938 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.281047 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:29.284195 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:29.284199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:29.284201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:29.284230 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:29.286170 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:29.287486 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:29.287489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:29.287491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:29.287617 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:29.288064 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:29.288067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:29.288069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:29.288313 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:29.288415 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:29.289201 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 18:33:29.289332 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2571,"thread_start_us":109,"threads_started":1}
I20260929 18:33:29.291087 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.266373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.1:40265)
0929 18:33:29.268857 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:29.268915 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:29.270002 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:29.276473 (+ 6471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:29.276487 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:29.276889 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:29.277664 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:29.277689 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.278935 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.278940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:29.280431 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:29.280442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.281236 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.281240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:29.281439 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:29.282166 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:29.282188 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:29.283968 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:29.286310 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:29.286317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:29.286333 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:29.287364 (+ 1031us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:29.287751 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:29.287754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:29.287756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:29.287941 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:29.288512 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:29.288518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:29.288891 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 18:33:29.289677 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 18:33:29.290031 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":136,"threads_started":1}
I20260929 18:33:29.300228 12829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:29.300588 12829 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:29.301265 12829 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:29.303272 12617 ts_manager.cc:194] Registered new tserver with Master: 13df9e05338744ebbabe9601ad09bbf8 (127.0.98.1:38799)
I20260929 18:33:29.304864 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:33:29.325591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:33:29.484061 12837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:29.484493 12837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:29.484608 12837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:29.490374 12837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:29.490442 12837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:29.490525 12837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:29.497860 12837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:29.499207 12837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:29.500525 12837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:29.511854 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:29.511861 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:29.512279 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:29.512810 12837 server_base.cc:1034] running on GCE node
I20260929 18:33:29.513356 12837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:29.514097 12837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:29.517402 12837 hybrid_clock.cc:648] HybridClock initialized: now 1790706809517370 us; error 44 us; skew 500 ppm
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:29.520969 12837 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:29.521948 12837 webserver.cc:460] Webserver started at http://127.0.98.2:38203/ using document root <none> and password file <none>
I20260929 18:33:29.522583 12837 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:29.522658 12837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:29.522862 12837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:29.539786 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "876b37916671a6064c37385d4c8e814c"
server_key_iv: "c1d08ba8e7f34fd4be4625002ee05d0d"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.540627 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "876b37916671a6064c37385d4c8e814c"
server_key_iv: "c1d08ba8e7f34fd4be4625002ee05d0d"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.545652 12837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 18:33:29.574769 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.576573 12837 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.001s
I20260929 18:33:29.576699 12837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "876b37916671a6064c37385d4c8e814c"
server_key_iv: "c1d08ba8e7f34fd4be4625002ee05d0d"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.576828 12837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:29.599936 12837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:29.603412 12837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:29.603642 12837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:29.604314 12837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:29.605304 12837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:29.605399 12837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.605489 12837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:29.605551 12837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.617204 12837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37283
I20260929 18:33:29.618469 12837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:29.618671 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37283 every 8 connection(s)
I20260929 18:33:29.626648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 12837
I20260929 18:33:29.626787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:29.627103 392 external_mini_cluster.cc:1428] Setting key ad411dbb4c5b8c2c661d127766a4ab66
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:29.637027 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.623019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49065 (local address 127.0.98.62:42549)
0929 18:33:29.623186 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:29.623191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:29.623718 (+ 527us) server_negotiation.cc:407] Connection header received
0929 18:33:29.625045 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:29.625049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:29.625102 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:29.625156 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:29.626927 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.627705 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.629473 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.629890 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.629980 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:29.632988 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:29.632992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:29.632995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:29.633022 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:29.634695 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:29.635296 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:29.635300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:29.635301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:29.635383 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:29.635763 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:29.635767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:29.635769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:29.635972 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:29.636052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:29.636726 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 18:33:29.636863 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:33:29.638753 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.621923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.2:49065)
0929 18:33:29.623553 (+ 1630us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:29.623608 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:29.624795 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:29.625608 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:29.625617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:29.625979 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:29.626636 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:29.626656 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.627939 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.627942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:29.629326 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:29.629340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:29.630184 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:29.630190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:29.630409 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:29.631058 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:29.631078 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:29.632804 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:29.634837 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:29.634843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:29.634858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:29.635178 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:29.635483 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:29.635486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:29.635489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:29.635646 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:29.636117 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:29.636123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:29.636454 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 18:33:29.637439 (+ 985us) client_negotiation.cc:240] Negotiation successful
0929 18:33:29.637762 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":137,"threads_started":1}
I20260929 18:33:29.640197 12967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:29.640376 12967 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:29.640908 12967 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:29.642032 12617 ts_manager.cc:194] Registered new tserver with Master: 3c5fdf65c8cb4e0c9b59036b66108bf6 (127.0.98.2:37283)
I20260929 18:33:29.642755 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49065
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:33:29.672971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:29.837110 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:29.837400 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:29.837532 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:29.843514 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:29.843590 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:29.843677 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:29.851274 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:29.852735 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:29.854106 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:29.866869 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:29.867029 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:29.867591 12972 server_base.cc:1034] running on GCE node
W20260929 18:33:29.868361 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:29.868844 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:29.869692 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:29.876541 12972 hybrid_clock.cc:648] HybridClock initialized: now 1790706809876509 us; error 42 us; skew 500 ppm
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:29.883217 12972 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:29.884634 12972 webserver.cc:460] Webserver started at http://127.0.98.3:34085/ using document root <none> and password file <none>
I20260929 18:33:29.885536 12972 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:29.885654 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:29.885901 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:33:29 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:29.911121 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "3401022bc9c64c3688907df9bf339a7c"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "81c4fdf1a9bfaafdd912d17f132a6591"
server_key_iv: "7952d6f7dd59c8b613905db2dfd41976"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.911945 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3401022bc9c64c3688907df9bf339a7c"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "81c4fdf1a9bfaafdd912d17f132a6591"
server_key_iv: "7952d6f7dd59c8b613905db2dfd41976"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.918758 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260929 18:33:29.940872 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.942610 12972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:33:29.942732 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "3401022bc9c64c3688907df9bf339a7c"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "81c4fdf1a9bfaafdd912d17f132a6591"
server_key_iv: "7952d6f7dd59c8b613905db2dfd41976"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:29.942881 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:29.959847 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:29.964545 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:29.965131 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:29.965711 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:29.966758 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:29.966862 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.966958 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:29.967020 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:29.978024 12972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33499
I20260929 18:33:29.979315 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:29.988991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 12972
I20260929 18:33:29.989147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:29.989508 392 external_mini_cluster.cc:1428] Setting key abeed7db839580d7f338fb5539004fbb
I20260929 18:33:29.991297 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33499 every 8 connection(s)
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:30.009382 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.994757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36605 (local address 127.0.98.62:42549)
0929 18:33:29.994868 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:29.994873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:29.996471 (+ 1598us) server_negotiation.cc:407] Connection header received
0929 18:33:29.997646 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:29.997649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:29.997700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:29.997749 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:29.999369 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.000243 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.001890 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.002239 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.002307 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.005363 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:30.005367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:30.005370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:30.005395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:30.007097 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.007683 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.007686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.007688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.007782 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.008173 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.008176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.008178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.008380 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:30.008448 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:30.009091 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 18:33:30.009225 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:33:30.011405 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:29.993656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.3:36605)
0929 18:33:29.996326 (+ 2670us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:29.996382 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:29.997415 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:29.998143 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:29.998162 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:29.998512 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:29.999213 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:29.999230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.000389 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.000393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.001756 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:30.001766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.002495 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.002502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.002707 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.003436 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:30.003457 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:30.005185 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:30.007243 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.007248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.007263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.007571 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.007889 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.007892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.007894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.008048 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.008526 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:30.008532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:30.008829 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 18:33:30.010108 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0929 18:33:30.010434 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2419,"thread_start_us":197,"threads_started":1}
I20260929 18:33:30.012822 13101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:30.013046 13101 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.013564 13101 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:30.014672 12616 ts_manager.cc:194] Registered new tserver with Master: 3401022bc9c64c3688907df9bf339a7c (127.0.98.3:33499)
I20260929 18:33:30.015439 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:30.015470 12616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36605
I20260929 18:33:30.016250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 12698
I20260929 18:33:30.025677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 12837
I20260929 18:33:30.033262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 12972
I20260929 18:33:30.040030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 12585
I20260929 18:33:30.044853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--webserver_interface=127.0.98.62
--webserver_port=40421
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42549
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:30.152557 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:30.152837 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:30.152910 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:30.157352 13104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:30.157433 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:30.157471 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:30.157524 13104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:30.157557 13104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:30.162974 13104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42549
--ranger_config_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:30.163910 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:30.164870 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:30.172830 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.172935 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.172964 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:30.173250 13104 server_base.cc:1034] running on GCE node
I20260929 18:33:30.173712 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:30.174685 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:30.175873 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790706810175846 us; error 39 us; skew 500 ppm
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:30.178611 13104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:30.179502 13104 webserver.cc:460] Webserver started at http://127.0.98.62:40421/ using document root <none> and password file <none>
I20260929 18:33:30.180042 13104 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:30.180107 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:30.226327 13104 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:33:30.228668 13120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.230018 13104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:33:30.230135 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0723843535114876bbc7c157e195f5d9"
format_stamp: "Formatted at 2026-09-29 18:33:28 on dist-test-slave-fmv8"
server_key: "4f66157c02ddfc7d97f6a8114eab8f5c"
server_key_iv: "5d0b916bbe8e527a576d80f7b98441da"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:30.230669 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:30.272388 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:30.274940 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:30.275098 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:30.282025 13104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42549
I20260929 18:33:30.282032 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42549 every 8 connection(s)
I20260929 18:33:30.283042 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:30.287151 13173 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:33:30.288730 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Bootstrap starting.
I20260929 18:33:30.291713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13104
I20260929 18:33:30.292136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38799
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42549
--webserver_port=32875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:30.305212 13173 log.cc:824] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:30.311683 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:30.312078 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Bootstrap complete.
I20260929 18:33:30.316255 13173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:30.316661 13173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0723843535114876bbc7c157e195f5d9, State: Initialized, Role: FOLLOWER
I20260929 18:33:30.317114 13173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:30.317250 13173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:30.317329 13173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:30.317437 13173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:33:30.319237 13173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:30.319566 13173 leader_election.cc:302] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0723843535114876bbc7c157e195f5d9; no voters:
I20260929 18:33:30.319883 13173 leader_election.cc:288] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:33:30.319967 13177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:33:30.320214 13177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [term 2 LEADER]: Becoming Leader. State: Replica: 0723843535114876bbc7c157e195f5d9, State: Running, Role: LEADER
I20260929 18:33:30.320554 13177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260929 18:33:30.321087 13173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:30.321992 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0723843535114876bbc7c157e195f5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260929 18:33:30.322000 13179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0723843535114876bbc7c157e195f5d9. Latest consensus state: current_term: 2 leader_uuid: "0723843535114876bbc7c157e195f5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0723843535114876bbc7c157e195f5d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260929 18:33:30.322122 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:30.322142 13179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:30.322576 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:30.323446 13173 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:33:30.328783 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:30.329469 13186 catalog_manager.cc:1259] Loaded cluster ID: dea9d6bc3c75449ea1f8c48ea2d31f52
I20260929 18:33:30.329561 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:30.332031 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:30.332772 13186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Loaded TSK: 0
I20260929 18:33:30.334018 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:33:30.437186 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:30.437493 13175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:30.437564 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:30.443637 13175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:30.443720 13175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:30.443811 13175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:30.451670 13175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32875
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:30.452952 13175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:30.454252 13175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:30.466212 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:30.466499 13175 server_base.cc:1034] running on GCE node
W20260929 18:33:30.466562 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.466747 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:30.467201 13175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:30.467916 13175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:30.469194 13175 hybrid_clock.cc:648] HybridClock initialized: now 1790706810469168 us; error 47 us; skew 500 ppm
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:30.472611 13175 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:30.473625 13175 webserver.cc:460] Webserver started at http://127.0.98.1:32875/ using document root <none> and password file <none>
I20260929 18:33:30.474411 13175 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:30.474558 13175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:30.497673 13175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:30.500360 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.501820 13175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:30.501962 13175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "13df9e05338744ebbabe9601ad09bbf8"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "6a116610ed482547c1ca71cbee13e456"
server_key_iv: "8f7ec1f27040abf26928f42ccaf175e1"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:30.502501 13175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:30.516268 13175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:30.519681 13175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:30.519868 13175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:30.520517 13175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:30.521564 13175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:30.521627 13175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.521698 13175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:30.521739 13175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.538463 13175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38799
I20260929 18:33:30.539297 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38799 every 8 connection(s)
I20260929 18:33:30.539825 13175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:30.546559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13175
I20260929 18:33:30.547060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42549
--webserver_port=38203
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:30.561326 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:30.542416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51693 (local address 127.0.98.62:42549)
0929 18:33:30.543682 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:30.543706 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:30.545431 (+ 1725us) server_negotiation.cc:407] Connection header received
0929 18:33:30.545757 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:30.545773 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:30.546171 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:30.546512 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:30.547958 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.549530 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.550784 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.551573 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.551764 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.555323 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:30.555328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:30.555340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:30.555364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:30.557491 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.558245 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.558255 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.558261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.558416 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.558826 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.558831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.558835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.559359 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:30.559527 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:30.560174 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 18:33:30.560454 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":139,"threads_started":1}
I20260929 18:33:30.561327 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:30.542537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.1:51693)
0929 18:33:30.543971 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:30.544024 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:30.545118 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:30.546962 (+ 1844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:30.546969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:30.547316 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:30.547790 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:30.547803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.549730 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.549733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.550658 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:30.550667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.551759 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.551767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.551966 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.552684 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:30.552707 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:30.554210 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:30.557647 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.557657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.557672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.558077 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.558534 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.558537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.558539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.558714 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.559493 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:30.559500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:30.559810 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 18:33:30.560013 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 18:33:30.560357 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":131,"threads_started":1}
I20260929 18:33:30.563912 13335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:30.564188 13335 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.564714 13335 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:30.566833 13335 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:33:30.566952 13335 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.567116 13335 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:30.567909 13335 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.568064 13335 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:30.568804 13335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:33:30.684187 13339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:30.684562 13339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:30.684659 13339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:30.691085 13339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:30.691174 13339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:30.691270 13339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:30.698827 13339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38203
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:30.699956 13339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:30.701112 13339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:30.713627 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.713577 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.714112 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:30.714390 13339 server_base.cc:1034] running on GCE node
I20260929 18:33:30.714854 13339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:30.715677 13339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:30.716876 13339 hybrid_clock.cc:648] HybridClock initialized: now 1790706810716842 us; error 48 us; skew 500 ppm
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:30.720280 13339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:30.721192 13339 webserver.cc:460] Webserver started at http://127.0.98.2:38203/ using document root <none> and password file <none>
I20260929 18:33:30.721922 13339 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:30.722033 13339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:30.743600 13339 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 18:33:30.746912 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.748677 13339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:33:30.748807 13339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "876b37916671a6064c37385d4c8e814c"
server_key_iv: "c1d08ba8e7f34fd4be4625002ee05d0d"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:30.749555 13339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:30.770740 13339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:30.774035 13339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:30.774243 13339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:30.774947 13339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:30.775990 13339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:30.776078 13339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:30.776181 13339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:30.776222 13339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 18:33:30.787382 13339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37283
I20260929 18:33:30.787674 13468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37283 every 8 connection(s)
I20260929 18:33:30.788623 13339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:30.789317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13339
I20260929 18:33:30.789788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33499
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42549
--webserver_port=34085
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:30.813143 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:30.794581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57521 (local address 127.0.98.62:42549)
0929 18:33:30.794745 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:30.794752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:30.799253 (+ 4501us) server_negotiation.cc:407] Connection header received
0929 18:33:30.800588 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:30.800591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:30.800653 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:30.800712 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:30.802406 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.803248 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.804955 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.805306 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.805388 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.808685 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:30.808689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:30.808692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:30.808726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:30.810535 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.811192 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.811196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.811198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.811288 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:30.811735 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:30.811741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:30.811743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:30.811972 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:30.812081 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:30.812815 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 18:33:30.812954 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:33:30.815017 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:30.793475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.2:57521)
0929 18:33:30.799068 (+ 5593us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:30.799130 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:30.800323 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:30.801150 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:30.801160 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:30.801522 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:30.802218 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:30.802240 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.803413 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.803416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.804815 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:30.804826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:30.805594 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:30.805598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:30.805795 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:30.806586 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:30.806607 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:30.808391 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:30.810673 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.810679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.810695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.811062 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.811411 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:30.811414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:30.811416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:30.811606 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:30.812174 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:30.812180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:30.812515 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 18:33:30.813607 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0929 18:33:30.813948 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5330,"thread_start_us":115,"threads_started":1}
I20260929 18:33:30.816680 13469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:30.816928 13469 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.817520 13469 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:30.819180 13469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:33:30.819300 13469 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.819458 13469 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:30.820344 13469 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:30.820518 13469 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:30.821238 13469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:33:30.951440 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:30.951769 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:30.951835 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:30.958091 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:30.958171 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:30.958267 13471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:30.966064 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:57099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34085
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:30.967404 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:30.968811 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:30.981426 13479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.981549 13481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:30.981441 13478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:30.982386 13471 server_base.cc:1034] running on GCE node
I20260929 18:33:30.982919 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:30.983791 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:30.985061 13471 hybrid_clock.cc:648] HybridClock initialized: now 1790706810985008 us; error 76 us; skew 500 ppm
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:30.992983 13471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:30.994218 13471 webserver.cc:460] Webserver started at http://127.0.98.3:34085/ using document root <none> and password file <none>
I20260929 18:33:30.994961 13471 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:30.995081 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:33:30 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:31.012266 13471 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:31.015674 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:31.017402 13471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:31.017601 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "3401022bc9c64c3688907df9bf339a7c"
format_stamp: "Formatted at 2026-09-29 18:33:29 on dist-test-slave-fmv8"
server_key: "81c4fdf1a9bfaafdd912d17f132a6591"
server_key_iv: "7952d6f7dd59c8b613905db2dfd41976"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:31.018440 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:31.053943 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:31.057677 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:31.057915 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:31.058625 13471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:31.059629 13471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:31.059721 13471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:31.059801 13471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:31.059851 13471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:31.072880 13471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33499
I20260929 18:33:31.073063 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33499 every 8 connection(s)
I20260929 18:33:31.073989 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:31.081015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13471
Sep 29 18:33:31 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:31.089286 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:31.076293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60885 (local address 127.0.98.62:42549)
0929 18:33:31.076429 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:31.076432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:31.077096 (+ 664us) server_negotiation.cc:407] Connection header received
0929 18:33:31.078051 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:31.078054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:31.078140 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:31.078233 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:31.079285 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.079813 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.081102 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.081460 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.081603 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:31.085262 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:31.085268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:31.085274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:31.085303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:31.086846 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:31.087518 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:31.087523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:31.087526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:31.087613 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:31.088028 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:31.088031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:31.088033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:31.088270 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:31.088373 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:31.089018 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 18:33:31.089155 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:33:31.090549 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:31.076445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.3:60885)
0929 18:33:31.076939 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:31.076981 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:31.077842 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:31.078401 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:31.078408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:31.078675 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:31.079142 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:31.079155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.079933 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.079937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:31.080954 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:31.080965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.081774 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.081778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:31.081971 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:31.083356 (+ 1385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:31.083375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:31.085094 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:31.086996 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:31.087002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:31.087018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:31.087398 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:31.087736 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:31.087742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:31.087747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:31.087926 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:31.088371 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:31.088378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:31.088721 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 18:33:31.089516 (+ 795us) client_negotiation.cc:240] Negotiation successful
0929 18:33:31.089801 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20260929 18:33:31.091651 13603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260929 18:33:31.091881 13603 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:31.092446 13603 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:31.094125 13603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42549 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:33:31.094245 13603 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:31.094398 13603 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:31.095232 13603 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:31.095393 13603 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
W20260929 18:33:31.096216 13603 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:33:31.097640 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:31.081662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.98.62:42549)
0929 18:33:31.081912 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:31.081922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:31.082154 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:33:31.082330 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:31.082334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:31.082383 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:31.082475 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:31.083256 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.084049 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.084954 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:31.085212 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:31.085308 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:31.086995 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:31.086998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:31.087001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:31.087024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:31.089055 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:31.096347 (+ 7292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:31.096350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:31.096352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:31.096433 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:31.096708 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:31.096712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:31.096714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:31.096887 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:31.097100 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:31.097251 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 18:33:31.097373 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
Sep 29 18:33:31 dist-test-slave-fmv8 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:31.569042 13335 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:31.569344 13335 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:31.570327 13137 ts_manager.cc:194] Registered new tserver with Master: 13df9e05338744ebbabe9601ad09bbf8 (127.0.98.1:38799)
I20260929 18:33:31.571715 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51693
I20260929 18:33:31.821444 13469 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:31.821727 13469 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:31.822471 13137 ts_manager.cc:194] Registered new tserver with Master: 3c5fdf65c8cb4e0c9b59036b66108bf6 (127.0.98.2:37283)
I20260929 18:33:31.823398 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57521
Sep 29 18:33:31 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 18:33:32 dist-test-slave-fmv8 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 18:33:32.096429 13603 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:32.096705 13603 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260929 18:33:32.097427 13136 ts_manager.cc:194] Registered new tserver with Master: 3401022bc9c64c3688907df9bf339a7c (127.0.98.3:33499)
I20260929 18:33:32.098186 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60885
I20260929 18:33:32.105688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:32.122334 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.107261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.98.62:42549)
0929 18:33:32.107606 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.107611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.109247 (+ 1636us) server_negotiation.cc:407] Connection header received
0929 18:33:32.109456 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.109459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.109509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.109569 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:32.112118 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.112994 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.114443 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.114800 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.114913 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.118511 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:32.118515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:32.118518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:32.118549 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:32.120485 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:32.120982 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:32.120985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:32.120987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:32.121079 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:32.121368 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:32.121371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:32.121373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:32.121594 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:32.121726 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.122010 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.122151 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":62,"threads_started":1}
I20260929 18:33:32.127466 13136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:33:32.132421 13136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:33:32.157675 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.148013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37283 (local address 127.0.0.1:45064)
0929 18:33:32.149656 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.149669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:32.149845 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:32.151007 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:32.151015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:32.151041 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:32.152014 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.152027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.153609 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.153613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.155085 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.155096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.157121 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.157125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.157240 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.157289 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 18:33:32.157381 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:33:32.157506 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1584,"mutex_wait_us":768,"thread_start_us":81,"threads_started":1}
I20260929 18:33:32.159451 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.148944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.0.98.2:37283)
0929 18:33:32.149214 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.149220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.149792 (+ 572us) server_negotiation.cc:407] Connection header received
0929 18:33:32.150565 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.150571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.150804 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.152225 (+ 1421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:32.152409 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.153478 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.156112 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.156959 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.158287 (+ 1328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.158449 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.158500 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.159297 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":112,"threads_started":1}
I20260929 18:33:32.162878 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b (DEFAULT_TABLE table=test-table [id=b7a05cf6b92f4c72a5da290aad727f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:32.163758 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.147705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38799 (local address 127.0.0.1:34558)
0929 18:33:32.148004 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.148055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:32.148223 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:32.152818 (+ 4595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:32.152821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:32.152839 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:32.152918 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.152924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.158468 (+ 5544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.158472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.159577 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.159588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.163359 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.163363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.163460 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.163497 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:33:32.163550 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:33:32.163636 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260929 18:33:32.164062 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1809ac8a489342ea9df3708a62bc5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:32.164590 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.146728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33499 (local address 127.0.0.1:56312)
0929 18:33:32.150193 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.150208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:32.150403 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:32.151119 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:32.151122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:32.151141 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:32.151263 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.151269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.161625 (+ 10356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.161628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.162710 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.162721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.164190 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.164197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.164303 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.164326 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:33:32.164396 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:33:32.164461 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260929 18:33:32.165010 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.148786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.98.1:38799)
0929 18:33:32.150107 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.150114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.150137 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:33:32.150219 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.150224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.152662 (+ 2438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.153065 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:32.153219 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.158338 (+ 5119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.161625 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.163218 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.164111 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.164723 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.164781 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.164878 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":67,"threads_started":1}
I20260929 18:33:32.165182 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.148336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.3:33499)
0929 18:33:32.148588 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.148598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.150328 (+ 1730us) server_negotiation.cc:407] Connection header received
0929 18:33:32.150569 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.150575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.150787 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.150910 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:32.151381 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.159601 (+ 8220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.162858 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.163619 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.163729 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.163860 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.164941 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.165050 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260929 18:33:32.168035 13262 tablet_service.cc:1467] Processing CreateTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b (DEFAULT_TABLE table=test-table [id=b7a05cf6b92f4c72a5da290aad727f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:32.169186 13262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1809ac8a489342ea9df3708a62bc5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:32.170861 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b (DEFAULT_TABLE table=test-table [id=b7a05cf6b92f4c72a5da290aad727f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:32.171818 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1809ac8a489342ea9df3708a62bc5e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:32.186175 13639 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Bootstrap starting.
I20260929 18:33:32.186602 13638 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Bootstrap starting.
I20260929 18:33:32.189009 13638 tablet_bootstrap.cc:654] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:32.191346 13639 tablet_bootstrap.cc:654] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:32.192521 13639 log.cc:824] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:32.193332 13638 log.cc:824] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:32.196211 13639 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: No bootstrap required, opened a new log
I20260929 18:33:32.196484 13639 ts_tablet_manager.cc:1397] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:33:32.196630 13640 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Bootstrap starting.
I20260929 18:33:32.198925 13640 tablet_bootstrap.cc:654] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:32.199728 13640 log.cc:824] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:32.200941 13639 raft_consensus.cc:348] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.201334 13639 raft_consensus.cc:374] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:32.202745 13638 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: No bootstrap required, opened a new log
I20260929 18:33:32.203003 13638 ts_tablet_manager.cc:1397] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260929 18:33:32.207145 13638 raft_consensus.cc:348] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.207497 13638 raft_consensus.cc:374] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:32.207597 13638 raft_consensus.cc:729] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5fdf65c8cb4e0c9b59036b66108bf6, State: Initialized, Role: FOLLOWER
I20260929 18:33:32.208225 13638 consensus_queue.cc:260] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.208472 13639 raft_consensus.cc:729] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3401022bc9c64c3688907df9bf339a7c, State: Initialized, Role: FOLLOWER
I20260929 18:33:32.209103 13640 tablet_bootstrap.cc:492] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: No bootstrap required, opened a new log
I20260929 18:33:32.209290 13640 ts_tablet_manager.cc:1397] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260929 18:33:32.209272 13639 consensus_queue.cc:260] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.210000 13603 heartbeater.cc:503] Master 127.0.98.62:42549 was elected leader, sending a full tablet report...
I20260929 18:33:32.210347 13639 ts_tablet_manager.cc:1428] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260929 18:33:32.213405 13640 raft_consensus.cc:348] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.213738 13640 raft_consensus.cc:374] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:32.213835 13640 raft_consensus.cc:729] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13df9e05338744ebbabe9601ad09bbf8, State: Initialized, Role: FOLLOWER
I20260929 18:33:32.214197 13638 ts_tablet_manager.cc:1428] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260929 18:33:32.214306 13640 consensus_queue.cc:260] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.214571 13469 heartbeater.cc:503] Master 127.0.98.62:42549 was elected leader, sending a full tablet report...
I20260929 18:33:32.215304 13640 ts_tablet_manager.cc:1428] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:33:32.215659 13335 heartbeater.cc:503] Master 127.0.98.62:42549 was elected leader, sending a full tablet report...
W20260929 18:33:32.292323 13336 tablet.cc:2369] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:33:32.295166 13470 tablet.cc:2369] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:33:32.325716 13604 tablet.cc:2369] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:33:32.445827 13644 raft_consensus.cc:482] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:32.446059 13644 raft_consensus.cc:504] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.447224 13644 leader_election.cc:288] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3401022bc9c64c3688907df9bf339a7c (127.0.98.3:33499), 13df9e05338744ebbabe9601ad09bbf8 (127.0.98.1:38799)
I20260929 18:33:32.452793 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.447609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38111 (local address 127.0.98.3:33499)
0929 18:33:32.447737 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.447743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.447991 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:33:32.448181 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.448184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.448231 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.448321 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:32.448684 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.449484 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.451418 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.452049 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.452180 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.452238 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.452626 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.452695 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:33:32.452908 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.447499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38799 (local address 127.0.98.2:33465)
0929 18:33:32.447976 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.447988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:32.448062 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:32.448359 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:32.448362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:32.448436 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:32.448613 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.448622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.449629 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.449633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.451278 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.451290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.452213 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.452217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.452487 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.452504 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:33:32.452577 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:33:32.452736 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":330,"spinlock_wait_cycles":38656,"thread_start_us":95,"threads_started":1}
I20260929 18:33:32.452912 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.447499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33499 (local address 127.0.98.2:38111)
0929 18:33:32.447889 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.447905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:32.448052 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:32.448359 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:32.448362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:32.448383 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:32.448552 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.448560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.449629 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.449633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.451278 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:32.451290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.452213 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.452217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:32.452459 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.452490 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:33:32.452577 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:33:32.452736 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":293,"thread_start_us":109,"threads_started":1}
I20260929 18:33:32.453161 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.447609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33465 (local address 127.0.98.1:38799)
0929 18:33:32.447740 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.447745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.448070 (+ 325us) server_negotiation.cc:407] Connection header received
0929 18:33:32.448148 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.448168 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.448231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.448316 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:32.448737 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.449514 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.451447 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.452035 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.452171 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.452237 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.452983 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.453058 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:33:32.453691 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1809ac8a489342ea9df3708a62bc5e8b" candidate_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13df9e05338744ebbabe9601ad09bbf8" is_pre_election: true
I20260929 18:33:32.453692 13543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1809ac8a489342ea9df3708a62bc5e8b" candidate_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3401022bc9c64c3688907df9bf339a7c" is_pre_election: true
I20260929 18:33:32.453986 13289 raft_consensus.cc:2454] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5fdf65c8cb4e0c9b59036b66108bf6 in term 0.
I20260929 18:33:32.454011 13543 raft_consensus.cc:2454] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5fdf65c8cb4e0c9b59036b66108bf6 in term 0.
I20260929 18:33:32.454478 13358 leader_election.cc:302] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13df9e05338744ebbabe9601ad09bbf8, 3401022bc9c64c3688907df9bf339a7c, 3c5fdf65c8cb4e0c9b59036b66108bf6; no voters:
I20260929 18:33:32.454748 13644 raft_consensus.cc:2781] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:32.454825 13644 raft_consensus.cc:482] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:32.454885 13644 raft_consensus.cc:3037] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:32.455973 13644 raft_consensus.cc:504] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.456321 13644 leader_election.cc:288] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 3401022bc9c64c3688907df9bf339a7c (127.0.98.3:33499), 13df9e05338744ebbabe9601ad09bbf8 (127.0.98.1:38799)
I20260929 18:33:32.456650 13543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1809ac8a489342ea9df3708a62bc5e8b" candidate_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3401022bc9c64c3688907df9bf339a7c"
I20260929 18:33:32.456779 13543 raft_consensus.cc:3037] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:32.457706 13543 raft_consensus.cc:2454] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5fdf65c8cb4e0c9b59036b66108bf6 in term 1.
I20260929 18:33:32.457991 13357 leader_election.cc:302] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3401022bc9c64c3688907df9bf339a7c, 3c5fdf65c8cb4e0c9b59036b66108bf6; no voters:
I20260929 18:33:32.457974 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1809ac8a489342ea9df3708a62bc5e8b" candidate_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13df9e05338744ebbabe9601ad09bbf8"
I20260929 18:33:32.458096 13289 raft_consensus.cc:3037] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:32.458153 13644 raft_consensus.cc:2781] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:32.458377 13644 raft_consensus.cc:686] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5fdf65c8cb4e0c9b59036b66108bf6, State: Running, Role: LEADER
I20260929 18:33:32.458724 13644 consensus_queue.cc:237] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } }
I20260929 18:33:32.459362 13289 raft_consensus.cc:2454] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5fdf65c8cb4e0c9b59036b66108bf6 in term 1.
I20260929 18:33:32.461630 13137 catalog_manager.cc:5495] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5fdf65c8cb4e0c9b59036b66108bf6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5fdf65c8cb4e0c9b59036b66108bf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37283 } health_report { overall_health: HEALTHY } } }
I20260929 18:33:32.517966 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.514632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.98.2:37283)
0929 18:33:32.514752 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:32.514755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:32.514769 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:33:32.514856 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:32.514858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:32.514915 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:32.515016 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:32.515222 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.515710 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.516664 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:32.516891 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:32.516968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:32.517220 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:32.517626 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:32.517715 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:32.517769 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 18:33:32.517817 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:33:32.523794 13543 raft_consensus.cc:1261] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 1 FOLLOWER]: Refusing update from remote peer 3c5fdf65c8cb4e0c9b59036b66108bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:32.523814 13289 raft_consensus.cc:1261] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8 [term 1 FOLLOWER]: Refusing update from remote peer 3c5fdf65c8cb4e0c9b59036b66108bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:32.524497 13644 consensus_queue.cc:1035] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3401022bc9c64c3688907df9bf339a7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:32.524880 13656 consensus_queue.cc:1035] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13df9e05338744ebbabe9601ad09bbf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:32.531566 13659 mvcc.cc:205] Tried to move back new op lower bound from 7334735104090222592 to 7334735103834238976. Current Snapshot: MvccSnapshot[applied={T|T < 7334735104090222592}]
I20260929 18:33:32.534132 13661 mvcc.cc:205] Tried to move back new op lower bound from 7334735104090222592 to 7334735103834238976. Current Snapshot: MvccSnapshot[applied={T|T < 7334735104090222592}]
I20260929 18:33:32.539131 13136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:32.539268 13136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:32.541462 13136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0723843535114876bbc7c157e195f5d9: Sending DeleteTablet for 3 replicas of tablet 1809ac8a489342ea9df3708a62bc5e8b
I20260929 18:33:32.542336 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56312
I20260929 18:33:32.542780 13669 tablet_replica.cc:330] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: stopping tablet replica
I20260929 18:33:32.542768 13262 tablet_service.cc:1514] Processing DeleteTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34558
I20260929 18:33:32.543160 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1809ac8a489342ea9df3708a62bc5e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45064
I20260929 18:33:32.543484 13669 raft_consensus.cc:2229] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:32.543829 13670 tablet_replica.cc:330] T 1809ac8a489342ea9df3708a62bc5e8b P 13df9e05338744ebbabe9601ad09bbf8: stopping tablet replica
I20260929 18:33:32.544240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13175
I20260929 18:33:32.546411 13669 raft_consensus.cc:2258] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:32.546788 13671 tablet_replica.cc:330] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: stopping tablet replica
I20260929 18:33:32.547112 13671 raft_consensus.cc:2229] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:33:32.547487 13671 raft_consensus.cc:2258] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:32.547991 13669 ts_tablet_manager.cc:1905] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:32.548914 13671 ts_tablet_manager.cc:1905] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:32.551011 13669 ts_tablet_manager.cc:1918] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:32.551183 13669 log.cc:1196] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/1809ac8a489342ea9df3708a62bc5e8b
I20260929 18:33:32.551594 13669 ts_tablet_manager.cc:1939] T 1809ac8a489342ea9df3708a62bc5e8b P 3401022bc9c64c3688907df9bf339a7c: Deleting consensus metadata
I20260929 18:33:32.551669 13671 ts_tablet_manager.cc:1918] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:32.551783 13671 log.cc:1196] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/1809ac8a489342ea9df3708a62bc5e8b
W20260929 18:33:32.551779 13123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38799 (error 108)
I20260929 18:33:32.552115 13671 ts_tablet_manager.cc:1939] T 1809ac8a489342ea9df3708a62bc5e8b P 3c5fdf65c8cb4e0c9b59036b66108bf6: Deleting consensus metadata
I20260929 18:33:32.552764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13339
I20260929 18:33:32.554567 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.554161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38799 (local address 127.0.0.1:34566)
0929 18:33:32.554375 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.554465 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 18:33:32.554736 13122 catalog_manager.cc:4841] TS 3401022bc9c64c3688907df9bf339a7c (127.0.98.3:33499): tablet 1809ac8a489342ea9df3708a62bc5e8b (table test-table [id=b7a05cf6b92f4c72a5da290aad727f4c]) successfully deleted
W20260929 18:33:32.554800 13123 catalog_manager.cc:4584] TS 13df9e05338744ebbabe9601ad09bbf8 (127.0.98.1:38799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1809ac8a489342ea9df3708a62bc5e8b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38799: connect: Connection refused (error 111)
I20260929 18:33:32.558645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13471
I20260929 18:33:32.559566 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:32.558847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37283 (local address 127.0.0.1:45078)
0929 18:33:32.559392 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:32.559462 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439}
I20260929 18:33:32.564327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13104
I20260929 18:33:32.570981 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:33:32.915261 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:33:32.915371 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:33:33.251663 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:33:33.251 UTC [12171] LOG: received smart shutdown request
2026-09-29 18:33:33.255 UTC [12171] LOG: background worker "logical replication launcher" (PID 12180) exited with exit code 1
2026-09-29 18:33:33.255 UTC [12175] LOG: shutting down
2026-09-29 18:33:33.403 UTC [12171] LOG: database system is shut down
2026-09-29T18:33:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48017 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:33 dist-test-slave-fmv8 krb5kdc[13681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:33 dist-test-slave-fmv8 krb5kdc[13681](info): set up 2 sockets
Sep 29 18:33:33 dist-test-slave-fmv8 krb5kdc[13681](info): commencing operation
krb5kdc: starting...
W20260929 18:33:34.556095 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:33:34 dist-test-slave-fmv8 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:33:34.674875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:34.782984 13694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:34.783236 13694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:34.783275 13694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:34.787237 13694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:34.787290 13694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:34.787314 13694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:34.787330 13694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:34.787369 13694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:34.792575 13694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:34.793547 13694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:34.794529 13694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:34.802892 13699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:34.802937 13702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:34.802906 13700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:34.803063 13694 server_base.cc:1034] running on GCE node
I20260929 18:33:34.803799 13694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:34.804886 13694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:34.806113 13694 hybrid_clock.cc:648] HybridClock initialized: now 1790706814806085 us; error 41 us; skew 500 ppm
Sep 29 18:33:34 dist-test-slave-fmv8 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:34.808959 13694 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:34.809801 13694 webserver.cc:460] Webserver started at http://127.0.98.62:46679/ using document root <none> and password file <none>
I20260929 18:33:34.810338 13694 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:34.810446 13694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:34.810660 13694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:34.812666 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "2d3f071b2a4a4c03b494d40f50f90498"
format_stamp: "Formatted at 2026-09-29 18:33:34 on dist-test-slave-fmv8"
server_key: "85d59d2a0130eb1aa8f16e394841a451"
server_key_iv: "5215cdaba24b442e32411409ebae8a89"
server_key_version: "encryptionkey@0"
I20260929 18:33:34.813213 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d3f071b2a4a4c03b494d40f50f90498"
format_stamp: "Formatted at 2026-09-29 18:33:34 on dist-test-slave-fmv8"
server_key: "85d59d2a0130eb1aa8f16e394841a451"
server_key_iv: "5215cdaba24b442e32411409ebae8a89"
server_key_version: "encryptionkey@0"
I20260929 18:33:34.816916 13694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:33:34.819285 13709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:34.820632 13694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:34.820732 13694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "2d3f071b2a4a4c03b494d40f50f90498"
format_stamp: "Formatted at 2026-09-29 18:33:34 on dist-test-slave-fmv8"
server_key: "85d59d2a0130eb1aa8f16e394841a451"
server_key_iv: "5215cdaba24b442e32411409ebae8a89"
server_key_version: "encryptionkey@0"
I20260929 18:33:34.820817 13694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:34.833532 13694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:34.836491 13694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:34.836639 13694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:34.843251 13694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46297
I20260929 18:33:34.843351 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46297 every 8 connection(s)
I20260929 18:33:34.844245 13694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:34.847090 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:34.850435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13694
I20260929 18:33:34.850531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:33:34.850826 392 external_mini_cluster.cc:1428] Setting key afffb7002b1ac13082db4413626b8e7b
I20260929 18:33:34.850946 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 18:33:34 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:34.867527 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.853230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.0.1:50394)
0929 18:33:34.853823 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.853859 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:34.854700 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:34.855365 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:34.855369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:34.855811 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:34.856036 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:34.856049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:34.857198 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:34.857201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:34.857909 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:34.857917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:34.858534 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:34.858538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:34.858640 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:34.859205 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:34.859222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:34.861918 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:34.864311 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:34.864319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:34.864327 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:34.864543 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:34.864988 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:34.864991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:34.864993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:34.865129 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:34.865930 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:34.865935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:34.866058 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 18:33:34.866201 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:33:34.866663 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":177,"spinlock_wait_cycles":73600,"thread_start_us":166,"threads_started":1}
I20260929 18:33:34.867537 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.853230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.98.62:46297)
0929 18:33:34.853656 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:34.853677 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:34.853959 (+ 282us) server_negotiation.cc:407] Connection header received
0929 18:33:34.855006 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:34.855011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:34.855188 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:34.855370 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:34.856229 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:34.857065 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:34.858025 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:34.858390 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:34.858552 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:34.862082 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:34.862087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:34.862097 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:34.862133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:34.864204 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:34.864670 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:34.864679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:34.864684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:34.864838 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:34.865241 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:34.865246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:34.865250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:34.865796 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:34.865951 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:34.866316 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 18:33:34.866600 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"thread_start_us":144,"threads_started":1}
I20260929 18:33:34.869903 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:33:34.870786 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.870533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54604)
0929 18:33:34.870666 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.870734 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:33:34.871032 13712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
I20260929 18:33:34.871937 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.871713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54620)
0929 18:33:34.871843 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.871889 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:33:34.872123 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 18:33:34.876966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46539
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:34.921409 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } attempt: 1
I20260929 18:33:34.922350 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.922027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54624)
0929 18:33:34.922194 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.922266 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:33:34.923237 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.923007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54636)
0929 18:33:34.923113 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.923171 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:33:34.923375 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
W20260929 18:33:34.987115 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:34.987537 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:34.987622 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:33:34.991546 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } attempt: 2
W20260929 18:33:34.991813 13769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:34.991878 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:34.991936 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:34.991966 13769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:34.992000 13769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:34.992493 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.992212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54646)
0929 18:33:34.992350 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.992415 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 18:33:34.993163 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:34.992960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54660)
0929 18:33:34.993059 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:34.993106 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:33:34.993331 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46539: connect: Connection refused (error 111)
I20260929 18:33:34.997705 13769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:34.998754 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:34.999749 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:35.008229 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:35.008301 13769 server_base.cc:1034] running on GCE node
W20260929 18:33:35.008250 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:35.008240 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:35.009027 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:35.010072 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:35.011265 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790706815011250 us; error 32 us; skew 500 ppm
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:35.014417 13769 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 18:33:35.015319 13769 webserver.cc:460] Webserver started at http://127.0.98.61:45925/ using document root <none> and password file <none>
I20260929 18:33:35.015892 13769 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:35.015978 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:35.016248 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:35.018236 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data/instance:
uuid: "481165aad4b242fcbd691b6dd4e852a0"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "4e368d2746e34dca83bff585d196dc84"
server_key_iv: "0da76b953d9ac4da94b2bb2b1601e6e8"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.018790 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal/instance:
uuid: "481165aad4b242fcbd691b6dd4e852a0"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "4e368d2746e34dca83bff585d196dc84"
server_key_iv: "0da76b953d9ac4da94b2bb2b1601e6e8"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.022357 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:33:35.024744 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:35.025929 13769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:35.026067 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "481165aad4b242fcbd691b6dd4e852a0"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "4e368d2746e34dca83bff585d196dc84"
server_key_iv: "0da76b953d9ac4da94b2bb2b1601e6e8"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.026170 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:35.043510 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:35.046811 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:35.047010 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:35.053886 13769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46539
I20260929 18:33:35.053910 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46539 every 8 connection(s)
I20260929 18:33:35.054764 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:33:35.057345 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:35.060963 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:33:35.062940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13769
I20260929 18:33:35.063073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-1/wal/instance
I20260929 18:33:35.063329 392 external_mini_cluster.cc:1428] Setting key 641ca70d6cc967e0a995dfaffbbcf6ae
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 18:33:35.076015 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.062701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.0.98.62:46297)
0929 18:33:35.062864 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.062869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.063745 (+ 876us) server_negotiation.cc:407] Connection header received
0929 18:33:35.064630 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.064636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.064707 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.064808 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.066392 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.066918 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.068382 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.068845 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.068982 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.071750 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.071753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.071755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.071779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.073956 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.074495 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.074501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.074503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.074582 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.074923 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.074928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.074930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.075109 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.075221 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.075719 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.075845 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:33:35.076824 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.063093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.0.1:50402)
0929 18:33:35.063568 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.063617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.064433 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.065016 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.065026 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.065510 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.066213 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.066233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.067052 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.067056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.068263 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.068273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.069021 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.069024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.069164 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.069823 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.069838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.071512 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.074066 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.074076 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.074094 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.074389 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.074698 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.074701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.074702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.074831 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.075221 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.075228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.075472 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.075661 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.075979 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260929 18:33:35.078250 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:33:35.084481 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } attempt: 3
I20260929 18:33:35.087903 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.078790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54662)
0929 18:33:35.078931 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.078941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.078997 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.079503 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.079506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.079675 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.079742 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.079747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.080732 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.080738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.081261 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.081268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.081837 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.081840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.081926 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.082371 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.082382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.083792 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.086169 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.086175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.086179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.086385 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.086723 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.086727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.086729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.086849 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.087344 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.087346 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.087435 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.087540 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.087709 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 18:33:35.088426 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.078890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.98.61:46539)
0929 18:33:35.079194 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.079203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.079220 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:33:35.079250 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.079254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.079394 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.079527 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.079842 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.080624 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.081366 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.081729 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.081838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.083934 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.083943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.083953 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.083974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.086026 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.086493 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.086499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.086500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.086590 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.086952 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.086955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.086957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.087237 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.087370 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.087636 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.087779 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260929 18:33:35.089115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41093
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:33:35.090404 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:33:35.091428 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.091176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45534)
0929 18:33:35.091279 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.091358 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 18:33:35.091610 13786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
I20260929 18:33:35.092881 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.092342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45550)
0929 18:33:35.092653 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.092717 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
W20260929 18:33:35.093142 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
I20260929 18:33:35.093876 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.084921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54672)
0929 18:33:35.085025 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.085041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.085136 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.085528 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.085531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.085717 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.085829 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.085835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.086828 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.086831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.087406 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.087415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.088364 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.088369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.088500 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.088965 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.088979 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.090231 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.092281 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.092284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.092286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.092535 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.092854 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.092857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.092859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.092949 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.093367 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.093370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.093471 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.093571 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.093724 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:33:35.093888 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.084969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.0.98.61:46539)
0929 18:33:35.085251 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.085296 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.085312 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:33:35.085351 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.085354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.085397 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.085492 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.085947 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.086732 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.087520 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.087897 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.088351 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.090386 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.090389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.090392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.090413 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.092099 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.092652 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.092655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.092657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.092731 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.093059 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.093062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.093063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.093280 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.093373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.093630 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.093732 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260929 18:33:35.094660 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:33:35.095316 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.095029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45566)
0929 18:33:35.095167 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.095232 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:33:35.096050 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.095761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45572)
0929 18:33:35.095917 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.095976 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 18:33:35.096251 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
I20260929 18:33:35.127410 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 1
I20260929 18:33:35.128317 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.127963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45588)
0929 18:33:35.128110 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.128237 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:33:35.129060 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.128876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45602)
0929 18:33:35.128970 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.129013 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:33:35.129230 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
I20260929 18:33:35.142407 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 1
I20260929 18:33:35.143198 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.142913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45618)
0929 18:33:35.143053 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.143110 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:33:35.143891 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.143688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45626)
0929 18:33:35.143793 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.143839 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:33:35.144008 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
W20260929 18:33:35.200167 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:35.200448 13844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:35.200485 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:35.204433 13844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260929 18:33:35.204371 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 2
W20260929 18:33:35.204483 13844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:35.204509 13844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:35.204530 13844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:35.204555 13844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:35.205164 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.204837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45640)
0929 18:33:35.205028 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.205090 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:33:35.205907 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.205728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45648)
0929 18:33:35.205816 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.205858 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:33:35.206056 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
I20260929 18:33:35.210022 13844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46297,127.0.98.61:46539,127.0.98.60:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:35.210975 13844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:35.211901 13844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:33:35.212198 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 2
I20260929 18:33:35.212823 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.212591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45660)
0929 18:33:35.212717 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.212766 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:33:35.213445 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.213251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45666)
0929 18:33:35.213349 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.213395 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:33:35.213563 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41093: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41093: connect: Connection refused (error 111)
W20260929 18:33:35.220031 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:35.220037 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:35.220037 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:35.220238 13844 server_base.cc:1034] running on GCE node
I20260929 18:33:35.220786 13844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:35.221777 13844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:35.222929 13844 hybrid_clock.cc:648] HybridClock initialized: now 1790706815222908 us; error 31 us; skew 500 ppm
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:35.225749 13844 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 18:33:35.226544 13844 webserver.cc:460] Webserver started at http://127.0.98.60:34077/ using document root <none> and password file <none>
I20260929 18:33:35.227062 13844 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:35.227146 13844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:35.227363 13844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:35.229413 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data/instance:
uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "b1399fe1f71e4fcb8a63b15ba67739bb"
server_key_iv: "eefd20c4c16576a849f3b57cba6bff99"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.230001 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal/instance:
uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "b1399fe1f71e4fcb8a63b15ba67739bb"
server_key_iv: "eefd20c4c16576a849f3b57cba6bff99"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.233572 13844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:33:35.235890 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:35.237010 13844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:35.237139 13844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3"
format_stamp: "Formatted at 2026-09-29 18:33:35 on dist-test-slave-fmv8"
server_key: "b1399fe1f71e4fcb8a63b15ba67739bb"
server_key_iv: "eefd20c4c16576a849f3b57cba6bff99"
server_key_version: "encryptionkey@0"
I20260929 18:33:35.237283 13844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:35.252904 13844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:35.257022 13844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:35.257236 13844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:35.264102 13844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41093
I20260929 18:33:35.264122 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41093 every 8 connection(s)
I20260929 18:33:35.265079 13844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:33:35.266010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 13844
I20260929 18:33:35.266134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706655775522-392-0/minicluster-data/master-2/wal/instance
I20260929 18:33:35.266435 392 external_mini_cluster.cc:1428] Setting key 9b13b5cbdd3465e1a0499b718c5d1391
I20260929 18:33:35.268443 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:35.272030 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:35.280292 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.267822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.0.98.60:41093)
0929 18:33:35.268285 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.268302 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.268337 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:33:35.269006 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.269023 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.269346 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.269672 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.270446 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.271403 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.272283 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.272791 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.272953 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.275662 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.275667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.275675 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.275701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.277380 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.277825 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.277830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.277834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.277941 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.278215 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.278218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.278220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.278737 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.278976 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.279201 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.279529 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260929 18:33:35.282886 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.273840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.98.62:46297)
0929 18:33:35.273983 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.273988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.274433 (+ 445us) server_negotiation.cc:407] Connection header received
0929 18:33:35.274570 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.274572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.274614 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.274810 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.275503 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.275987 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.276887 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.277101 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.277211 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.279422 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.279425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.279427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.279450 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.281035 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.281591 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.281594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.281595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.281673 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.281941 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.281944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.281945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.282177 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.282268 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.282593 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.282732 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:33:35.282912 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.273764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46297 (local address 127.0.0.1:50408)
0929 18:33:35.274284 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.274335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.274471 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.274857 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.274862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.275255 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.275382 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.275387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.276098 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.276101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.276767 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.276776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.277206 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.277209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.277316 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.277790 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.277801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.279235 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.281155 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.281163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.281167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.281474 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.281755 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.281758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.281760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.281848 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.282279 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.282287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.282388 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.282511 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.282710 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"spinlock_wait_cycles":2816,"thread_start_us":140,"threads_started":1}
I20260929 18:33:35.284183 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:33:35.292284 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.282880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.0.98.62:46297)
0929 18:33:35.283120 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.283124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.283136 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:33:35.283169 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.283172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.283214 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.283342 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.284035 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.284751 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.285708 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.286021 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.286119 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.288590 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.288593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.288596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.288622 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.290334 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.290814 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.290832 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.290836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.290923 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.291262 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.291269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.291273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.291512 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.291737 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.291920 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.292064 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":125,"threads_started":1}
I20260929 18:33:35.293331 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.284629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46539 (local address 127.0.0.1:54682)
0929 18:33:35.284761 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.284776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.284848 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.285070 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.285073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.285229 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.285328 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.285333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.286282 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.286287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.287142 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.287156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.287774 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.287776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.287842 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.288319 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.288331 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.289629 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.291641 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.291645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.291646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.291913 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.292268 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.292271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.292272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.292347 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.292822 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.292825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.292922 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.293048 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.293190 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:33:35.293558 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.284734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.98.61:46539)
0929 18:33:35.284872 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.284876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.284904 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:33:35.284939 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.284941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.284981 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.285035 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.285434 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.286159 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.287341 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.287662 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.287773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.289790 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.289798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.289802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.289832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.291534 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.292009 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.292015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.292018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.292100 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.292497 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.292503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.292506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.292719 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.292806 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.293110 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.293290 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:33:35.294128 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:33:35.303247 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.293616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.0.98.61:46539)
0929 18:33:35.293766 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.293770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.293958 (+ 188us) server_negotiation.cc:407] Connection header received
0929 18:33:35.294197 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.294199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.294260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.294345 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.294986 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.295505 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.296263 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.296483 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.296589 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.299292 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.299295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.299297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.299334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.301382 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.301824 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.301827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.301828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.301905 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.302213 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.302216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.302218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.302508 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.302732 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.302903 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.303030 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:33:35.303342 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.294653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.0.98.60:41093)
0929 18:33:35.294755 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.294759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.294771 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:33:35.294845 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.294848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.294878 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.294976 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.295305 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.295788 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.296551 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.297267 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.297377 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.299836 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.299840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.299843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.299870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.301587 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.302036 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.302039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.302040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.302103 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.302394 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.302398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.302400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.302658 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.302752 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.303041 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.303178 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":44}
I20260929 18:33:35.303321 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.294594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45680)
0929 18:33:35.294684 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.294699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.294767 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.294987 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.294989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.295138 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.295203 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.295208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.295900 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.295903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.296450 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.296458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.297376 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.297380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.297454 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.298003 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.298014 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.299646 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.301701 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.301704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.301706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.301927 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.302209 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.302212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.302214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.302292 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.302769 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.302773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.302872 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.302976 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.303112 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 18:33:35.304996 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 3
I20260929 18:33:35.305191 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } attempt: 3
I20260929 18:33:35.306931 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: Bootstrap starting.
I20260929 18:33:35.307289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:33:35.309670 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:35.310597 13914 log.cc:824] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:33:35.312935 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: No bootstrap required, opened a new log
I20260929 18:33:35.316109 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.305525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45686)
0929 18:33:35.305768 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.305788 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.305864 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.306676 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.306680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.306858 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.306974 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.306982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.308053 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.308058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.308888 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.308900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.310025 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.310028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.310121 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.310713 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.310731 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.312278 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.314418 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.314422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.314424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.314697 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.315034 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.315038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.315041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.315134 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.315583 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.315586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.315683 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.315785 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.315929 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 18:33:35.316622 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.306048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.98.60:41093)
0929 18:33:35.306443 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.306451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.306464 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:33:35.306496 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.306499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.306538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.306624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.307112 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.307923 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.309083 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.309419 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.309521 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.312421 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.312425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.312427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.312449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.314292 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.314824 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.314828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.314830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.314918 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.315219 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.315224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.315226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.315425 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.315531 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.316338 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.316455 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":109,"threads_started":1}
I20260929 18:33:35.317217 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.317490 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:35.317548 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1ca8c2f9064db8b1f8d735a437a3b3, State: Initialized, Role: FOLLOWER
Sep 29 18:33:35 dist-test-slave-fmv8 krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:33:35.318089 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.319639 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:35.321105 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: Bootstrap starting.
I20260929 18:33:35.322119 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.305496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41093 (local address 127.0.0.1:45684)
0929 18:33:35.311048 (+ 5552us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:35.311063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:35.311159 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:35.311418 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:35.311421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:35.311636 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:35.311768 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.311775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.313000 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.313003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.314071 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:35.314081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.314716 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.314719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:35.314820 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.316798 (+ 1978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:35.316812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:35.318300 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:35.320274 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.320278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.320280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.320559 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.320893 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:35.320896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:35.320900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:35.321004 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:35.321499 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:35.321504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:35.321620 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 18:33:35.321743 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:33:35.321886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5477}
I20260929 18:33:35.322628 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.305993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.0.98.60:41093)
0929 18:33:35.306276 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.306281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.311242 (+ 4961us) server_negotiation.cc:407] Connection header received
0929 18:33:35.311280 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.311283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.311330 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.311400 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.311897 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.312622 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.314202 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.314502 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.314595 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.318450 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.318454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.318456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.318477 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.320109 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.320686 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.320692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.320695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.320782 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.321121 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.321125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.321126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.321344 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.321446 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.322382 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.322494 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260929 18:33:35.324071 13762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:35.324993 13762 log.cc:824] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:35.325989 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: Bootstrap starting.
I20260929 18:33:35.323541 13844 master_runner.cc:186] Error getting master registration for 127.0.98.62:46297: OK, instance_id { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" instance_seqno: 1790706814841373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:33:35.326927 13844 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 18:33:35.327051 13941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:35.327101 13941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:35.327239 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.327312 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.328392 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.309460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.0.98.62:46297)
0929 18:33:35.309567 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.309572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.309801 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:33:35.309984 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.309990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.310036 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.310099 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.311071 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.311887 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.313059 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.313419 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.313498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.315545 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.315548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.315551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.315576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.319885 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.326093 (+ 6208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.326096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.326098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.326181 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.326791 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.326795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.326797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.326996 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.327123 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.328114 (+ 991us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.328255 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:33:35.328544 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:35.330199 13837 log.cc:824] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:35.331426 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: No bootstrap required, opened a new log
I20260929 18:33:35.331892 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.310436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.0.98.60:41093)
0929 18:33:35.310696 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.310706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.313327 (+ 2621us) server_negotiation.cc:407] Connection header received
0929 18:33:35.315304 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.315310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.315360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.315419 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.316591 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.317733 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.319118 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.319520 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.319612 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.321744 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.321751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.321754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.321777 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.329850 (+ 8073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.330390 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.330396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.330399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.330489 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.330845 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.330850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.330855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.331069 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.331203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.331571 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.331696 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260929 18:33:35.332095 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: No bootstrap required, opened a new log
I20260929 18:33:35.334789 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:35.309840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.0.98.61:46539)
0929 18:33:35.309956 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:35.309962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:35.326010 (+ 16048us) server_negotiation.cc:407] Connection header received
0929 18:33:35.326591 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:35.326595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:35.326637 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:35.326684 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:35.327820 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.328498 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.329648 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:35.329959 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:35.330064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:35.331676 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:35.331680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:35.331683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:35.331704 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:35.333244 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.333627 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.333632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.333633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.333711 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:35.333973 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:35.333976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:35.333978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:35.334138 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:35.334260 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:35.334479 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 18:33:35.334604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:33:35.335666 13762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.335883 13762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:35.335929 13762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3f071b2a4a4c03b494d40f50f90498, State: Initialized, Role: FOLLOWER
I20260929 18:33:35.336443 13762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.337769 13762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:35.337765 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
W20260929 18:33:35.337972 13715 tablet.cc:2369] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:33:35.338052 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:35.338127 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481165aad4b242fcbd691b6dd4e852a0, State: Initialized, Role: FOLLOWER
I20260929 18:33:35.338618 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.341946 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:33:35.343315 13957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:35.343400 13957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:35.342667 13694 master_runner.cc:186] Error getting master registration for 127.0.98.61:46539: OK, instance_id { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" instance_seqno: 1790706815051830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:33:35.347941 13694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 18:33:35.348215 13969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:35.348328 13969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:35.349583 13769 master_runner.cc:263] No leader master found from master 481165aad4b242fcbd691b6dd4e852a0
I20260929 18:33:35.349645 13769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:33:35.352313 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.352504 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.356292 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.356467 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.417062 13930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:35.417263 13930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.418807 13736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3f071b2a4a4c03b494d40f50f90498" is_pre_election: true
I20260929 18:33:35.419117 13736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1ca8c2f9064db8b1f8d735a437a3b3 in term 0.
I20260929 18:33:35.419549 13865 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3f071b2a4a4c03b494d40f50f90498, 7f1ca8c2f9064db8b1f8d735a437a3b3; no voters:
I20260929 18:33:35.423040 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481165aad4b242fcbd691b6dd4e852a0" is_pre_election: true
I20260929 18:33:35.423323 13811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1ca8c2f9064db8b1f8d735a437a3b3 in term 0.
I20260929 18:33:35.423666 13930 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d3f071b2a4a4c03b494d40f50f90498 (127.0.98.62:46297), 481165aad4b242fcbd691b6dd4e852a0 (127.0.98.61:46539)
I20260929 18:33:35.423956 13930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:35.424032 13930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:35.424079 13930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:35.425491 13930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.426067 13736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3f071b2a4a4c03b494d40f50f90498"
I20260929 18:33:35.426213 13736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:35.427465 13736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1ca8c2f9064db8b1f8d735a437a3b3 in term 1.
I20260929 18:33:35.427776 13865 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3f071b2a4a4c03b494d40f50f90498, 7f1ca8c2f9064db8b1f8d735a437a3b3; no voters:
I20260929 18:33:35.428308 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481165aad4b242fcbd691b6dd4e852a0"
I20260929 18:33:35.428431 13811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:35.429668 13811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1ca8c2f9064db8b1f8d735a437a3b3 in term 1.
I20260929 18:33:35.430020 13930 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d3f071b2a4a4c03b494d40f50f90498 (127.0.98.62:46297), 481165aad4b242fcbd691b6dd4e852a0 (127.0.98.61:46539)
I20260929 18:33:35.430160 13930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:35.442859 13930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1ca8c2f9064db8b1f8d735a437a3b3, State: Running, Role: LEADER
I20260929 18:33:35.443363 13930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } }
I20260929 18:33:35.449685 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1ca8c2f9064db8b1f8d735a437a3b3. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.449841 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:35.451822 13973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:35.452883 13973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:35.458333 13811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [term 1 FOLLOWER]: Refusing update from remote peer 7f1ca8c2f9064db8b1f8d735a437a3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:35.459030 13930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:35.463358 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1ca8c2f9064db8b1f8d735a437a3b3. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.463510 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.464612 13736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [term 1 FOLLOWER]: Refusing update from remote peer 7f1ca8c2f9064db8b1f8d735a437a3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:35.467746 13930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:35.471889 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1ca8c2f9064db8b1f8d735a437a3b3. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.472021 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.474792 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.474913 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:35.476191 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.476310 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.476785 13974 mvcc.cc:205] Tried to move back new op lower bound from 7334735116109750272 to 7334735116060921856. Current Snapshot: MvccSnapshot[applied={T|T < 7334735116109750272}]
I20260929 18:33:35.483780 13973 catalog_manager.cc:1347] Generated new cluster ID: 50392ef4a1c648a19b775e949f4eb6a0
I20260929 18:33:35.483844 13973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:35.499666 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.499809 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:33:35.499900 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d3f071b2a4a4c03b494d40f50f90498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46297 } } peers { permanent_uuid: "481165aad4b242fcbd691b6dd4e852a0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46539 } } peers { permanent_uuid: "7f1ca8c2f9064db8b1f8d735a437a3b3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41093 } } }
I20260929 18:33:35.500011 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:35.500627 13977 mvcc.cc:205] Tried to move back new op lower bound from 7334735116109750272 to 7334735116060921856. Current Snapshot: MvccSnapshot[applied={T|T < 7334735116109750272}]
I20260929 18:33:35.509070 13973 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:35.510214 13973 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:35.520288 13973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1ca8c2f9064db8b1f8d735a437a3b3: Generated new TSK 0
I20260929 18:33:35.521268 13973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:33:36.344800 13957 catalog_manager.cc:1259] Loaded cluster ID: 50392ef4a1c648a19b775e949f4eb6a0
I20260929 18:33:36.344875 13957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: loading cluster ID for follower catalog manager: success
I20260929 18:33:36.346557 13957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: acquiring CA information for follower catalog manager: success
I20260929 18:33:36.347779 13957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d3f071b2a4a4c03b494d40f50f90498: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:33:36.349462 13969 catalog_manager.cc:1259] Loaded cluster ID: 50392ef4a1c648a19b775e949f4eb6a0
I20260929 18:33:36.349516 13969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: loading cluster ID for follower catalog manager: success
I20260929 18:33:36.351033 13969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: acquiring CA information for follower catalog manager: success
I20260929 18:33:36.352087 13969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 481165aad4b242fcbd691b6dd4e852a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:33:36.594388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13694
I20260929 18:33:36.601112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13769
I20260929 18:33:36.609112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 13844
2026-09-29T18:33:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3085 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:36 dist-test-slave-fmv8 krb5kdc[13988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:36 dist-test-slave-fmv8 krb5kdc[13988](info): set up 2 sockets
Sep 29 18:33:36 dist-test-slave-fmv8 krb5kdc[13988](info): commencing operation
krb5kdc: starting...
W20260929 18:33:37.648336 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:33:37 dist-test-slave-fmv8 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:33:37.760532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:37.866917 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:37.867214 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:37.867292 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:37.871266 14001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:37.871361 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:37.871395 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:37.871438 14001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:37.871469 14001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:37.876772 14001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:37.877777 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:37.878842 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:37.887148 14007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:37.887305 14001 server_base.cc:1034] running on GCE node
W20260929 18:33:37.887176 14009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:37.887178 14006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:37.887944 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:37.889056 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:37.890244 14001 hybrid_clock.cc:648] HybridClock initialized: now 1790706817890230 us; error 27 us; skew 500 ppm
Sep 29 18:33:37 dist-test-slave-fmv8 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:37.892871 14001 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:37.893697 14001 webserver.cc:460] Webserver started at http://127.0.98.62:44283/ using document root <none> and password file <none>
I20260929 18:33:37.894229 14001 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:37.894317 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:37.894548 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:37.896566 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "88d5d04307824b60926d72a58281672d"
format_stamp: "Formatted at 2026-09-29 18:33:37 on dist-test-slave-fmv8"
server_key: "18298e04ad0a9e2683b9640977eb4721"
server_key_iv: "e9791eb4258ae84ab574e907a41ab750"
server_key_version: "encryptionkey@0"
I20260929 18:33:37.897119 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "88d5d04307824b60926d72a58281672d"
format_stamp: "Formatted at 2026-09-29 18:33:37 on dist-test-slave-fmv8"
server_key: "18298e04ad0a9e2683b9640977eb4721"
server_key_iv: "e9791eb4258ae84ab574e907a41ab750"
server_key_version: "encryptionkey@0"
I20260929 18:33:37.900781 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:33:37.903281 14016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:37.904563 14001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:37.904703 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "88d5d04307824b60926d72a58281672d"
format_stamp: "Formatted at 2026-09-29 18:33:37 on dist-test-slave-fmv8"
server_key: "18298e04ad0a9e2683b9640977eb4721"
server_key_iv: "e9791eb4258ae84ab574e907a41ab750"
server_key_version: "encryptionkey@0"
I20260929 18:33:37.904814 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:37.915546 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:37.918530 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:37.918720 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:37.925571 14001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43373
I20260929 18:33:37.925593 14068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43373 every 8 connection(s)
I20260929 18:33:37.926549 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:37.929246 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:37.934487 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: Bootstrap starting.
I20260929 18:33:37.936019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14001
I20260929 18:33:37.936134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:33:37.936442 392 external_mini_cluster.cc:1428] Setting key 3203a42e8720b40ca9934e235dc16d0b
I20260929 18:33:37.937139 14069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:37.938019 14069 log.cc:824] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:37.939793 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: No bootstrap required, opened a new log
I20260929 18:33:37.943198 14069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260929 18:33:37.943451 14069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:37.943531 14069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d5d04307824b60926d72a58281672d, State: Initialized, Role: FOLLOWER
I20260929 18:33:37.943938 14069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260929 18:33:37.944097 14069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:37.944199 14069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:37.944319 14069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:33:37 dist-test-slave-fmv8 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:37.945499 14069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260929 18:33:37.945839 14069 leader_election.cc:302] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d5d04307824b60926d72a58281672d; no voters:
I20260929 18:33:37.946170 14069 leader_election.cc:288] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:37.946293 14074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:37.946589 14074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [term 1 LEADER]: Becoming Leader. State: Replica: 88d5d04307824b60926d72a58281672d, State: Running, Role: LEADER
I20260929 18:33:37.946995 14074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260929 18:33:37.947477 14069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:37.949033 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d5d04307824b60926d72a58281672d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } } }
I20260929 18:33:37.949162 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:37.949026 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d5d04307824b60926d72a58281672d. Latest consensus state: current_term: 1 leader_uuid: "88d5d04307824b60926d72a58281672d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d5d04307824b60926d72a58281672d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } } }
I20260929 18:33:37.949831 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:37.951661 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:37.937471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.62:43373)
0929 18:33:37.937887 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:37.937902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:37.937936 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:33:37.938615 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:37.938632 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:37.938955 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:37.939284 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:37.940090 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:37.941104 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:37.941952 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:37.942406 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:37.942582 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:37.945762 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:37.945767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:37.945780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:37.945805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:37.947914 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:37.948453 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:37.948465 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:37.948470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:37.948631 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:37.948995 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:37.948999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:37.949001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:37.949513 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:37.949709 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:37.950471 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 18:33:37.950765 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
W20260929 18:33:37.953102 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:37.953189 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:37.953269 14083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:37.954052 14083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:37.954847 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:33:37.959671 14083 catalog_manager.cc:1347] Generated new cluster ID: 6759b09bbdcf49e6bc3d381a75c88918
I20260929 18:33:37.959738 14083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:37.967792 14083 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:37.968983 14083 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:37.982856 14083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d5d04307824b60926d72a58281672d: Generated new TSK 0
I20260929 18:33:37.983718 14083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:33:38.010864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14001
2026-09-29T18:33:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:38 dist-test-slave-fmv8 krb5kdc[14096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:38 dist-test-slave-fmv8 krb5kdc[14096](info): set up 2 sockets
Sep 29 18:33:38 dist-test-slave-fmv8 krb5kdc[14096](info): commencing operation
krb5kdc: starting...
W20260929 18:33:39.040329 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:33:39.160596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:39.269678 14109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:39.269936 14109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:39.269977 14109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:39.273981 14109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:39.274039 14109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:39.274057 14109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:39.274075 14109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:39.274089 14109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:39.279493 14109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:39.280555 14109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:39.281586 14109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:39.289945 14114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.289947 14115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.289968 14117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:39.290035 14109 server_base.cc:1034] running on GCE node
I20260929 18:33:39.290706 14109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:39.291812 14109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:39.293008 14109 hybrid_clock.cc:648] HybridClock initialized: now 1790706819292968 us; error 62 us; skew 500 ppm
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:39.295930 14109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:39.296903 14109 webserver.cc:460] Webserver started at http://127.0.98.62:40483/ using document root <none> and password file <none>
I20260929 18:33:39.297475 14109 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:39.297536 14109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:39.297744 14109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:39.299767 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "4ec3f6ef90414ca7ae17097398fc294f"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b88cdf02b9c06374667393d9cda5970f"
server_key_iv: "a7c05421588f8b107b667c2b013c5baf"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.300361 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ec3f6ef90414ca7ae17097398fc294f"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b88cdf02b9c06374667393d9cda5970f"
server_key_iv: "a7c05421588f8b107b667c2b013c5baf"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.304090 14109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:33:39.306656 14124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.307930 14109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:33:39.308079 14109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4ec3f6ef90414ca7ae17097398fc294f"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b88cdf02b9c06374667393d9cda5970f"
server_key_iv: "a7c05421588f8b107b667c2b013c5baf"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.308250 14109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:39.320436 14109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:39.323268 14109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:39.323467 14109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:39.330497 14109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38557
I20260929 18:33:39.330830 14176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38557 every 8 connection(s)
I20260929 18:33:39.331482 14109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:39.334237 14177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:39.336941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14109
I20260929 18:33:39.337065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:33:39.337384 392 external_mini_cluster.cc:1428] Setting key 92a6f52893ea495e4c59b9f3e78fbd25
I20260929 18:33:39.340131 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap starting.
I20260929 18:33:39.342612 14177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:39.343607 14177 log.cc:824] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:39.345623 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: No bootstrap required, opened a new log
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:39.348865 14177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:39.349112 14177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:39.349192 14177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Initialized, Role: FOLLOWER
I20260929 18:33:39.349642 14177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:39.349799 14177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:39.349908 14177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:39.350030 14177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:39.350943 14177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:39.351336 14177 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec3f6ef90414ca7ae17097398fc294f; no voters:
I20260929 18:33:39.351718 14177 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:39.351814 14182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:39.352011 14182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 LEADER]: Becoming Leader. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Running, Role: LEADER
I20260929 18:33:39.352415 14182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:39.352517 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:39.338679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.98.62:38557)
0929 18:33:39.339113 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:39.339128 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:39.339162 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:33:39.339762 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:39.339777 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:39.340090 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:39.340460 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:39.341284 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.342284 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.343638 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.344238 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.344416 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:39.346880 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:39.346886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:39.346899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:39.346924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:39.349280 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.349763 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.349768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.349775 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.349951 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.350269 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.350273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.350275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.350659 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:39.350863 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:39.351165 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:33:39.351460 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
I20260929 18:33:39.353102 14177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:39.353799 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec3f6ef90414ca7ae17097398fc294f. Latest consensus state: current_term: 1 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:39.353930 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:39.354231 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:39.354367 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
W20260929 18:33:39.357859 14197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:39.357931 14197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:39.358012 14190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:39.358701 14190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:39.364190 14190 catalog_manager.cc:1347] Generated new cluster ID: ab9b646acc414956ada398233b6efe67
I20260929 18:33:39.364250 14190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:39.379853 14190 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:39.380820 14190 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:39.386153 14190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Generated new TSK 0
I20260929 18:33:39.386816 14190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:33:39.421247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:39.556191 14203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:39.556464 14203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:39.556540 14203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:39.560542 14203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:39.560595 14203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:39.560649 14203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:39.565629 14203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:39.566553 14203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:39.567564 14203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:39.576884 14209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.576876 14208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.576895 14211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:39.577299 14203 server_base.cc:1034] running on GCE node
I20260929 18:33:39.577672 14203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:39.578275 14203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:39.579481 14203 hybrid_clock.cc:648] HybridClock initialized: now 1790706819579462 us; error 30 us; skew 500 ppm
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:39.582208 14203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:39.583112 14203 webserver.cc:460] Webserver started at http://127.0.98.1:43783/ using document root <none> and password file <none>
I20260929 18:33:39.583639 14203 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:39.583727 14203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:39.583948 14203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:39.585978 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "aec27c80404e45f48714fd33ad0726ca"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "13a11e7cdda411f85e6200b8a3f996cb"
server_key_iv: "ed326adb72363b86c6051064cf8e344f"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.586519 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aec27c80404e45f48714fd33ad0726ca"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "13a11e7cdda411f85e6200b8a3f996cb"
server_key_iv: "ed326adb72363b86c6051064cf8e344f"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.590147 14203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:33:39.592507 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.593670 14203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:39.593808 14203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "aec27c80404e45f48714fd33ad0726ca"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "13a11e7cdda411f85e6200b8a3f996cb"
server_key_iv: "ed326adb72363b86c6051064cf8e344f"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.593914 14203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:39.613888 14203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:39.616863 14203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:39.617054 14203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:39.617681 14203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:39.618476 14203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:39.618543 14203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.618614 14203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:39.618656 14203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.627784 14203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40215
I20260929 18:33:39.627803 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40215 every 8 connection(s)
I20260929 18:33:39.628680 14203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:39.637966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14203
I20260929 18:33:39.638095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:39.638441 392 external_mini_cluster.cc:1428] Setting key 398b3456f78e3bd274482a9289d3bce1
I20260929 18:33:39.642212 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:39.630467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60067 (local address 127.0.98.62:38557)
0929 18:33:39.630599 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:39.630603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:39.631621 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 18:33:39.632559 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:39.632563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:39.632619 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:39.632697 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:39.633810 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.634361 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.635492 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.635733 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.635847 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:39.638369 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:39.638375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:39.638379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:39.638409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:39.640140 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.640720 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.640726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.640728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.640811 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.641148 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.641153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.641155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.641331 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:39.641431 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:39.641932 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:33:39.642049 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:33:39.643242 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:39.630846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:60067)
0929 18:33:39.631474 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:39.631514 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:39.632353 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:39.632864 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:39.632871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:39.633206 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:39.633653 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:39.633665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.634489 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.634493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:39.635367 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:39.635376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.635881 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.635885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:39.636051 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:39.636601 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:39.636617 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:39.638180 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:39.640319 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:39.640326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:39.640343 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:39.640610 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:39.640902 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:39.640907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:39.640909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:39.641052 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:39.641442 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:39.641450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:39.641687 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:33:39.641869 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:33:39.642180 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":109,"threads_started":1}
I20260929 18:33:39.644563 14332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:39.644810 14332 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:39.645263 14332 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:39.646747 14141 ts_manager.cc:194] Registered new tserver with Master: aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:33:39.647900 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:33:39.664816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:39.772570 14337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:39.772823 14337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:39.772898 14337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:39.777010 14337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:39.777065 14337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:39.777120 14337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:39.782316 14337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:39.783219 14337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:39.784245 14337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:39.793747 14343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.793792 14345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:39.793771 14342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:39.794081 14337 server_base.cc:1034] running on GCE node
I20260929 18:33:39.794497 14337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:39.795082 14337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:39.796248 14337 hybrid_clock.cc:648] HybridClock initialized: now 1790706819796226 us; error 27 us; skew 500 ppm
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:39.799171 14337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:39.800063 14337 webserver.cc:460] Webserver started at http://127.0.98.2:37483/ using document root <none> and password file <none>
I20260929 18:33:39.800644 14337 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:39.800755 14337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:39.800976 14337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:39.802999 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b21885c85ac4a52a5c26425ad447dbe"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b1ffe6e0a871d18ff1bd322e18f84ddc"
server_key_iv: "d3923ab9fa89ede8a3f4e457e8726702"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.803553 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b21885c85ac4a52a5c26425ad447dbe"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b1ffe6e0a871d18ff1bd322e18f84ddc"
server_key_iv: "d3923ab9fa89ede8a3f4e457e8726702"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.807168 14337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:33:39.809505 14352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.810822 14337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:33:39.810966 14337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "1b21885c85ac4a52a5c26425ad447dbe"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b1ffe6e0a871d18ff1bd322e18f84ddc"
server_key_iv: "d3923ab9fa89ede8a3f4e457e8726702"
server_key_version: "encryptionkey@0"
I20260929 18:33:39.811077 14337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:39.823982 14337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:39.827121 14337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:39.827325 14337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:39.827941 14337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:39.828786 14337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:39.828864 14337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.828926 14337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:39.828976 14337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:39.837800 14337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43061
I20260929 18:33:39.837836 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43061 every 8 connection(s)
I20260929 18:33:39.838635 14337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:39.840324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14337
I20260929 18:33:39.840445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:39.840736 392 external_mini_cluster.cc:1428] Setting key 9bd5ccca825bfba5db97180432d267f6
Sep 29 18:33:39 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:33:39.853808 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:39.840446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43359 (local address 127.0.98.62:38557)
0929 18:33:39.840589 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:39.840592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:39.841898 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 18:33:39.842844 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:39.842848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:39.842901 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:39.842987 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:39.844116 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.844690 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.846478 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.846706 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.846797 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:39.849696 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:39.849700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:39.849702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:39.849726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:39.851340 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.851875 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.851879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.851881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.851964 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:39.852388 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:39.852391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:39.852393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:39.852655 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:39.852765 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:39.853384 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 18:33:39.853557 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:33:39.854861 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:39.840975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:43359)
0929 18:33:39.841729 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:39.841780 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:39.842633 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:39.843173 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:39.843180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:39.843520 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:39.843973 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:39.843985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.844822 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.844828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:39.846347 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:39.846362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:39.846873 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:39.846880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:39.847106 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:39.847762 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:39.847788 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:39.849517 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:39.851483 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:39.851488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:39.851504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:39.851760 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:39.852092 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:39.852095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:39.852097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:39.852276 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:39.852799 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:39.852807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:39.853060 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:33:39.853326 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 18:33:39.853643 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":131,"threads_started":1}
I20260929 18:33:39.856235 14466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:39.856530 14466 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:39.857046 14466 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:39.858132 14141 ts_manager.cc:194] Registered new tserver with Master: 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2:43061)
I20260929 18:33:39.858722 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:33:39.867599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:39.979818 14471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:39.980115 14471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:39.980222 14471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:39.984503 14471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:39.984567 14471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:39.984647 14471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:39.990397 14471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:39.991514 14471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:39.992630 14471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:40.002498 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.002681 14471 server_base.cc:1034] running on GCE node
W20260929 18:33:40.002614 14477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:40.002579 14476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.003181 14471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:40.003866 14471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:40.005023 14471 hybrid_clock.cc:648] HybridClock initialized: now 1790706820004983 us; error 50 us; skew 500 ppm
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:40.007664 14471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:40.008556 14471 webserver.cc:460] Webserver started at http://127.0.98.3:38371/ using document root <none> and password file <none>
I20260929 18:33:40.009096 14471 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:40.009183 14471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:40.009428 14471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:40.011483 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c53d71b8ab04fd381075b330e4c4265"
format_stamp: "Formatted at 2026-09-29 18:33:40 on dist-test-slave-fmv8"
server_key: "de026c1d127c15c2f7dcad22cdda3cf4"
server_key_iv: "ef08d56f1054388d5f15b685fb834f82"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.012035 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c53d71b8ab04fd381075b330e4c4265"
format_stamp: "Formatted at 2026-09-29 18:33:40 on dist-test-slave-fmv8"
server_key: "de026c1d127c15c2f7dcad22cdda3cf4"
server_key_iv: "ef08d56f1054388d5f15b685fb834f82"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.015714 14471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:40.018110 14486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.019491 14471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:40.019649 14471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "2c53d71b8ab04fd381075b330e4c4265"
format_stamp: "Formatted at 2026-09-29 18:33:40 on dist-test-slave-fmv8"
server_key: "de026c1d127c15c2f7dcad22cdda3cf4"
server_key_iv: "ef08d56f1054388d5f15b685fb834f82"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.019769 14471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:40.042990 14471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:40.046427 14471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:40.046631 14471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:40.047231 14471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:40.048072 14471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:40.048146 14471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.048229 14471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:40.048302 14471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.057578 14471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40085
I20260929 18:33:40.057613 14599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40085 every 8 connection(s)
I20260929 18:33:40.058441 14471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:40.064076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14471
I20260929 18:33:40.064240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:40.064524 392 external_mini_cluster.cc:1428] Setting key f428463738563fe8ddf68708e7f016de
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:40.072132 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.060223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47617 (local address 127.0.98.62:38557)
0929 18:33:40.060346 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.060349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.061446 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 18:33:40.062379 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.062383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.062441 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.062543 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:40.063772 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.064444 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.065743 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.066006 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.066101 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.068620 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:40.068623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:40.068625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:40.068649 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:40.070215 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.070721 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.070724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.070725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.070804 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.071120 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.071123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.071124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.071305 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:40.071399 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.071903 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.072014 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:33:40.072984 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.060613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:47617)
0929 18:33:40.061286 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.061327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.062158 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.062760 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.062767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.063127 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:40.063616 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.063635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.064569 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.064572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.065613 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.065630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.066156 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.066159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.066321 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.066959 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:40.066975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:40.068449 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:40.070317 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.070321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.070332 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.070567 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.070897 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.070899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.070901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.071026 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.071440 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:40.071445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:40.071671 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.071842 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.072095 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":113,"threads_started":1}
I20260929 18:33:40.074206 14600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:40.074422 14600 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:40.074900 14600 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:40.075923 14141 ts_manager.cc:194] Registered new tserver with Master: 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085)
I20260929 18:33:40.076545 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47617
I20260929 18:33:40.078604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:40.088130 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.080602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.98.62:38557)
0929 18:33:40.080729 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.080732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.080992 (+ 260us) server_negotiation.cc:407] Connection header received
0929 18:33:40.081137 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.081140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.081182 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.081274 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:40.081968 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.082488 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.083344 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.083574 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.083685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.085137 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:40.085141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:40.085144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:40.085168 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:40.086421 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.086858 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.086862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.086864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.086966 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.087245 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.087248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.087250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.087475 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:40.087619 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.087859 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.087973 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:33:40.091037 14141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:33:40.094465 14141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:33:40.108615 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.103157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.2:43061)
0929 18:33:40.103686 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.103698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.104041 (+ 343us) server_negotiation.cc:407] Connection header received
0929 18:33:40.104130 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.104136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.104393 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.104553 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:40.104834 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.105837 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.107319 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.107927 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.108037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.108232 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.108376 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.108466 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":127,"threads_started":1}
I20260929 18:33:40.108623 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.102964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43061 (local address 127.0.0.1:38292)
0929 18:33:40.103829 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.103844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.103946 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.104539 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.104544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.104571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:40.104724 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.104731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.105965 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.105970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.107196 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.107206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.108047 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.108051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.108188 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.108231 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.108347 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.108467 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":94,"threads_started":1}
I20260929 18:33:40.109082 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.102849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.0.1:41034)
0929 18:33:40.103277 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.103321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.103640 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.104936 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.104939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.104957 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:40.105031 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.105036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.106423 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.106426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.107630 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.107640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.108752 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.108757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.108850 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.108875 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.108924 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.108994 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":77,"threads_started":1}
I20260929 18:33:40.109539 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.104006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.0.1:55122)
0929 18:33:40.104272 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.104284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.104382 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.105287 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.105289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.105307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:40.105383 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.105388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.106664 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.106667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.107782 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.107790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.109124 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.109131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.109233 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.109258 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.109334 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.109409 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260929 18:33:40.109763 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.103620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.0.98.1:40215)
0929 18:33:40.104522 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.104537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.104557 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:33:40.104627 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.104631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.104807 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.104959 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:40.105105 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.106278 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.107782 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.108612 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.109373 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.109502 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.109547 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.109626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":93,"threads_started":1}
I20260929 18:33:40.109973 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.104451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.0.98.3:40085)
0929 18:33:40.104875 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.104888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.104911 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:33:40.104979 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.104984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.105152 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.105296 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:40.105482 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.106531 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.107912 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.108843 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.108940 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.109059 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.109706 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.109812 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260929 18:33:40.112380 14266 tablet_service.cc:1467] Processing CreateTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a (DEFAULT_TABLE table=test-table [id=c330c3728e114751bf702b4ad64997c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:40.112396 14400 tablet_service.cc:1467] Processing CreateTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a (DEFAULT_TABLE table=test-table [id=c330c3728e114751bf702b4ad64997c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:40.113433 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f2f77f333b45fe9cf9742cd5a8335a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:40.113471 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f2f77f333b45fe9cf9742cd5a8335a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:40.113181 14534 tablet_service.cc:1467] Processing CreateTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a (DEFAULT_TABLE table=test-table [id=c330c3728e114751bf702b4ad64997c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:40.114039 14534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f2f77f333b45fe9cf9742cd5a8335a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:40.119522 14617 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Bootstrap starting.
I20260929 18:33:40.121027 14619 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Bootstrap starting.
I20260929 18:33:40.121971 14617 tablet_bootstrap.cc:654] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:40.123013 14619 tablet_bootstrap.cc:654] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:40.123091 14617 log.cc:824] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:40.123534 14618 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Bootstrap starting.
I20260929 18:33:40.123921 14619 log.cc:824] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:40.125517 14617 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: No bootstrap required, opened a new log
I20260929 18:33:40.125697 14618 tablet_bootstrap.cc:654] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:40.125735 14617 ts_tablet_manager.cc:1397] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:40.126012 14619 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: No bootstrap required, opened a new log
I20260929 18:33:40.126211 14619 ts_tablet_manager.cc:1397] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:33:40.126896 14618 log.cc:824] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:40.128839 14618 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: No bootstrap required, opened a new log
I20260929 18:33:40.129014 14618 ts_tablet_manager.cc:1397] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:40.128835 14617 raft_consensus.cc:348] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.129120 14617 raft_consensus.cc:374] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:40.129208 14617 raft_consensus.cc:729] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec27c80404e45f48714fd33ad0726ca, State: Initialized, Role: FOLLOWER
I20260929 18:33:40.129721 14617 consensus_queue.cc:260] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.130065 14619 raft_consensus.cc:348] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.130362 14619 raft_consensus.cc:374] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:40.130396 14332 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 18:33:40.130425 14619 raft_consensus.cc:729] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Initialized, Role: FOLLOWER
I20260929 18:33:40.130849 14619 consensus_queue.cc:260] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.131669 14617 ts_tablet_manager.cc:1428] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:40.131762 14619 ts_tablet_manager.cc:1428] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:33:40.132135 14466 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 18:33:40.132722 14618 raft_consensus.cc:348] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.133015 14618 raft_consensus.cc:374] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:40.133121 14618 raft_consensus.cc:729] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c53d71b8ab04fd381075b330e4c4265, State: Initialized, Role: FOLLOWER
I20260929 18:33:40.133546 14618 consensus_queue.cc:260] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.134132 14600 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 18:33:40.134438 14618 ts_tablet_manager.cc:1428] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 18:33:40.213004 14624 raft_consensus.cc:482] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:40.213212 14624 raft_consensus.cc:504] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.214506 14624 leader_election.cc:288] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215), 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085)
I20260929 18:33:40.218557 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.214766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.98.2:52665)
0929 18:33:40.214985 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.215001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.215144 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.215494 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.215497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.215517 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:40.215653 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.215660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.216570 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.216574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.217405 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.217414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.218121 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.218127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.218231 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.218274 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.218347 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.218404 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 18:33:40.218623 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.214873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52665 (local address 127.0.98.1:40215)
0929 18:33:40.215008 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.215014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.215128 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:33:40.215282 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.215289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.215360 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.215443 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:40.215774 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.216466 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.217539 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.218002 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.218089 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.218182 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.218384 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.218476 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:33:40.218679 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.214876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39481 (local address 127.0.98.3:40085)
0929 18:33:40.215008 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.215014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.215365 (+ 351us) server_negotiation.cc:407] Connection header received
0929 18:33:40.215510 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.215514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.215565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.215642 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:40.215972 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.216556 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.217665 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.218071 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.218170 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.218240 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.218471 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.218536 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:33:40.218752 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.214772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.98.2:39481)
0929 18:33:40.215252 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.215267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.215401 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.215668 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.215671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.215686 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:40.215831 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.215838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.216674 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.216677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.217541 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.217551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.218202 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.218209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.218310 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.218360 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.218447 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.218512 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":43,"thread_start_us":130,"threads_started":1}
I20260929 18:33:40.219370 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f2f77f333b45fe9cf9742cd5a8335a" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c53d71b8ab04fd381075b330e4c4265" is_pre_election: true
I20260929 18:33:40.219416 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f2f77f333b45fe9cf9742cd5a8335a" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca" is_pre_election: true
I20260929 18:33:40.219689 14554 raft_consensus.cc:2454] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 0.
I20260929 18:33:40.219692 14286 raft_consensus.cc:2454] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 0.
I20260929 18:33:40.220201 14353 leader_election.cc:302] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, 2c53d71b8ab04fd381075b330e4c4265, aec27c80404e45f48714fd33ad0726ca; no voters:
I20260929 18:33:40.220507 14624 raft_consensus.cc:2781] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:40.220587 14624 raft_consensus.cc:482] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:40.220629 14624 raft_consensus.cc:3037] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:40.221591 14624 raft_consensus.cc:504] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.221909 14624 leader_election.cc:288] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 election: Requested vote from peers aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215), 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085)
I20260929 18:33:40.222209 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f2f77f333b45fe9cf9742cd5a8335a" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca"
I20260929 18:33:40.222342 14286 raft_consensus.cc:3037] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:40.222357 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f2f77f333b45fe9cf9742cd5a8335a" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c53d71b8ab04fd381075b330e4c4265"
I20260929 18:33:40.222466 14554 raft_consensus.cc:3037] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:40.223647 14554 raft_consensus.cc:2454] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 1.
I20260929 18:33:40.223647 14286 raft_consensus.cc:2454] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 1.
I20260929 18:33:40.223961 14354 leader_election.cc:302] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, 2c53d71b8ab04fd381075b330e4c4265; no voters:
I20260929 18:33:40.224145 14624 raft_consensus.cc:2781] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:40.224383 14624 raft_consensus.cc:686] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 1 LEADER]: Becoming Leader. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Running, Role: LEADER
I20260929 18:33:40.224740 14624 consensus_queue.cc:237] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.227705 14140 catalog_manager.cc:5495] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b21885c85ac4a52a5c26425ad447dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } health_report { overall_health: UNKNOWN } } }
I20260929 18:33:40.263253 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.255788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.0.98.2:43061)
0929 18:33:40.255924 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.255929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.255975 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:33:40.256219 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.256221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.256287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.256403 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:40.256652 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.257148 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.258363 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.258800 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.258916 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.262129 (+ 3213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:40.262743 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:40.262891 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.262987 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.263063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:33:40.269954 14286 raft_consensus.cc:1261] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Refusing update from remote peer 1b21885c85ac4a52a5c26425ad447dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:40.269954 14554 raft_consensus.cc:1261] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Refusing update from remote peer 1b21885c85ac4a52a5c26425ad447dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:40.270679 14627 consensus_queue.cc:1035] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:40.270871 14624 consensus_queue.cc:1035] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:40.278931 14632 mvcc.cc:205] Tried to move back new op lower bound from 7334735135818838016 to 7334735135643205632. Current Snapshot: MvccSnapshot[applied={T|T < 7334735135818838016}]
I20260929 18:33:40.279134 14631 mvcc.cc:205] Tried to move back new op lower bound from 7334735135818838016 to 7334735135643205632. Current Snapshot: MvccSnapshot[applied={T|T < 7334735135818838016}]
I20260929 18:33:40.286203 14140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:40.286350 14140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:40.288839 14140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Sending DeleteTablet for 3 replicas of tablet 03f2f77f333b45fe9cf9742cd5a8335a
I20260929 18:33:40.289613 14266 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41034
I20260929 18:33:40.289649 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38292
I20260929 18:33:40.289680 14534 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55122
I20260929 18:33:40.290091 14639 tablet_replica.cc:330] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: stopping tablet replica
I20260929 18:33:40.290256 14641 tablet_replica.cc:330] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: stopping tablet replica
I20260929 18:33:40.290282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14203
I20260929 18:33:40.290335 14640 tablet_replica.cc:330] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: stopping tablet replica
I20260929 18:33:40.290656 14641 raft_consensus.cc:2229] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:40.290673 14640 raft_consensus.cc:2229] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:40.290977 14640 raft_consensus.cc:2258] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:40.291405 14639 raft_consensus.cc:2229] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:33:40.291800 14639 raft_consensus.cc:2258] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:40.293306 14639 ts_tablet_manager.cc:1905] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:40.293306 14640 ts_tablet_manager.cc:1905] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:40.296087 14639 ts_tablet_manager.cc:1918] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:40.296284 14639 log.cc:1196] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/03f2f77f333b45fe9cf9742cd5a8335a
I20260929 18:33:40.296689 14640 ts_tablet_manager.cc:1918] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:40.296712 14639 ts_tablet_manager.cc:1939] T 03f2f77f333b45fe9cf9742cd5a8335a P 1b21885c85ac4a52a5c26425ad447dbe: Deleting consensus metadata
I20260929 18:33:40.296882 14640 log.cc:1196] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/03f2f77f333b45fe9cf9742cd5a8335a
I20260929 18:33:40.297269 14640 ts_tablet_manager.cc:1939] T 03f2f77f333b45fe9cf9742cd5a8335a P 2c53d71b8ab04fd381075b330e4c4265: Deleting consensus metadata
I20260929 18:33:40.297935 14126 catalog_manager.cc:4841] TS 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2:43061): tablet 03f2f77f333b45fe9cf9742cd5a8335a (table test-table [id=c330c3728e114751bf702b4ad64997c0]) successfully deleted
I20260929 18:33:40.298717 14126 catalog_manager.cc:4841] TS 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085): tablet 03f2f77f333b45fe9cf9742cd5a8335a (table test-table [id=c330c3728e114751bf702b4ad64997c0]) successfully deleted
W20260929 18:33:40.299424 14125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40215 (error 108)
I20260929 18:33:40.299615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14337
I20260929 18:33:40.301092 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.300751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.0.1:41046)
0929 18:33:40.300913 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.300992 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 18:33:40.301255 14125 catalog_manager.cc:4584] TS aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03f2f77f333b45fe9cf9742cd5a8335a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40215: connect: Connection refused (error 111)
I20260929 18:33:40.305980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14471
I20260929 18:33:40.311719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14109
I20260929 18:33:40.318240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--webserver_interface=127.0.98.62
--webserver_port=40483
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:40.426395 14643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:40.426710 14643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:40.426796 14643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:40.430801 14643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:40.430895 14643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:40.430935 14643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:40.430972 14643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:40.431005 14643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:40.436120 14643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:40.437157 14643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:40.438174 14643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:40.451064 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.451316 14643 server_base.cc:1034] running on GCE node
W20260929 18:33:40.451064 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:40.451079 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.452080 14643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:40.453313 14643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:40.454509 14643 hybrid_clock.cc:648] HybridClock initialized: now 1790706820454487 us; error 31 us; skew 500 ppm
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:40.457394 14643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:40.458345 14643 webserver.cc:460] Webserver started at http://127.0.98.62:40483/ using document root <none> and password file <none>
I20260929 18:33:40.458930 14643 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:40.459041 14643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:40.462893 14643 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:40.466261 14659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.467811 14643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:40.467948 14643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4ec3f6ef90414ca7ae17097398fc294f"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b88cdf02b9c06374667393d9cda5970f"
server_key_iv: "a7c05421588f8b107b667c2b013c5baf"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.468634 14643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:40.486790 14643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:40.489635 14643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:40.489848 14643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:40.497042 14643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38557
I20260929 18:33:40.497323 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38557 every 8 connection(s)
I20260929 18:33:40.498126 14643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:40.502367 14712 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:33:40.503890 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap starting.
I20260929 18:33:40.504359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14643
I20260929 18:33:40.504766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40215
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=43783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:40.520712 14712 log.cc:824] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:40.529148 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:40.529541 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap complete.
I20260929 18:33:40.533655 14712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:40.534032 14712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Initialized, Role: FOLLOWER
I20260929 18:33:40.534555 14712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:40.534670 14712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:40.534759 14712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:40.534871 14712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:33:40.536759 14712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:40.537084 14712 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec3f6ef90414ca7ae17097398fc294f; no voters:
I20260929 18:33:40.537380 14712 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:33:40.537505 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:33:40.537766 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 LEADER]: Becoming Leader. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Running, Role: LEADER
I20260929 18:33:40.538163 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:40.538600 14712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:40.539551 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec3f6ef90414ca7ae17097398fc294f. Latest consensus state: current_term: 2 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:40.539680 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:40.540669 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:40.541109 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:40.543498 14733 catalog_manager.cc:1259] Loaded cluster ID: ab9b646acc414956ada398233b6efe67
I20260929 18:33:40.543557 14733 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: loading cluster ID for follower catalog manager: success
W20260929 18:33:40.544505 14733 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260929 18:33:40.544576 14733 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260929 18:33:40.544627 14725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:40.546880 14725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:40.547389 14725 catalog_manager.cc:1259] Loaded cluster ID: ab9b646acc414956ada398233b6efe67
I20260929 18:33:40.547448 14725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:40.548514 14725 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:33:40.553372 14725 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:33:40.558524 14725 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:40.559218 14725 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Loaded TSK: 0
I20260929 18:33:40.560384 14725 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:33:40.619192 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:40.619462 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:40.619542 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:40.623622 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:40.623838 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:40.623986 14714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:40.629462 14714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43783
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:40.630554 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:40.631561 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:40.640846 14741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:40.640923 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:40.640949 14743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.641718 14714 server_base.cc:1034] running on GCE node
I20260929 18:33:40.642115 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:40.642772 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:40.643968 14714 hybrid_clock.cc:648] HybridClock initialized: now 1790706820643942 us; error 44 us; skew 500 ppm
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:40.646903 14714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:40.647856 14714 webserver.cc:460] Webserver started at http://127.0.98.1:43783/ using document root <none> and password file <none>
I20260929 18:33:40.648452 14714 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:40.648561 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:40.652392 14714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:33:40.654428 14750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.655643 14714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:40.655732 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "aec27c80404e45f48714fd33ad0726ca"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "13a11e7cdda411f85e6200b8a3f996cb"
server_key_iv: "ed326adb72363b86c6051064cf8e344f"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.656316 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:40.675951 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:40.679271 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:40.679482 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:40.680194 14714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:40.681423 14757 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:33:40.685564 14714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:33:40.685628 14714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 18:33:40.685678 14714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:33:40.687306 14714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:33:40.687361 14714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:33:40.687608 14757 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Bootstrap starting.
I20260929 18:33:40.698199 14714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40215
I20260929 18:33:40.698251 14864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40215 every 8 connection(s)
I20260929 18:33:40.699396 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:40.701112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14714
I20260929 18:33:40.701574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43061
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=37483
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:40.708997 14757 log.cc:824] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:40.718199 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.701445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:52241)
0929 18:33:40.702394 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.702449 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.703693 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.705098 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.705105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.705442 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:40.705897 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.705910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.707966 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.707973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.709640 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.709654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.710709 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.710713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.710899 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.711480 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:40.711497 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:40.713114 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:40.715173 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.715182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.715195 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.715469 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.715815 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.715819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.715821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.715958 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.716575 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:40.716581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:40.716837 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.717013 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.717331 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":140,"threads_started":1}
I20260929 18:33:40.718189 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.701440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52241 (local address 127.0.98.62:38557)
0929 18:33:40.702069 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.702090 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.702565 (+ 475us) server_negotiation.cc:407] Connection header received
0929 18:33:40.704256 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.704272 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.704679 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.705069 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:40.706071 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.707782 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.709798 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.710553 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.710767 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.713310 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:40.713315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:40.713327 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:40.713353 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:40.715047 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.715591 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.715600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.715604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.715718 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.716053 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.716058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.716059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.716459 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:40.716604 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.717133 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.717399 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":109,"threads_started":1}
I20260929 18:33:40.718334 14757 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:40.718747 14757 tablet_bootstrap.cc:492] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Bootstrap complete.
I20260929 18:33:40.720201 14757 ts_tablet_manager.cc:1397] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20260929 18:33:40.721498 14865 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:40.721725 14865 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:40.722172 14865 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:40.723006 14757 raft_consensus.cc:348] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.723588 14757 raft_consensus.cc:729] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec27c80404e45f48714fd33ad0726ca, State: Initialized, Role: FOLLOWER
I20260929 18:33:40.724085 14757 consensus_queue.cc:260] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } } peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260929 18:33:40.724586 14676 ts_manager.cc:194] Registered new tserver with Master: aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
I20260929 18:33:40.725085 14757 ts_tablet_manager.cc:1428] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:33:40.727892 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52241
I20260929 18:33:40.730485 14865 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:33:40.737756 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.727224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.0.1:41056)
0929 18:33:40.727722 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.727773 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.727896 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.728495 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.728501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.728940 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:40.729070 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.729078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.730379 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.730383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.731289 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.731300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.732013 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.732028 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.732131 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.732700 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:40.732714 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:40.734159 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:40.736000 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.736008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.736013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.736312 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.736601 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.736605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.736607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.736698 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.737126 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:40.737131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:40.737253 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.737383 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.737567 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":126,"threads_started":1}
I20260929 18:33:40.737754 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.727321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.1:40215)
0929 18:33:40.727952 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.727965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.728053 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:33:40.728119 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.728124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.728337 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.728471 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:40.729221 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.730259 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.731428 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.731926 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.732030 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.734299 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:40.734313 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:40.734322 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:40.734346 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:40.735870 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.736418 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.736424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.736426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.736507 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.736784 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.736788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.736790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.737011 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:40.737150 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.737463 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.737612 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":119,"threads_started":1}
I20260929 18:33:40.738756 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f2f77f333b45fe9cf9742cd5a8335a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41056
I20260929 18:33:40.739677 14874 tablet_replica.cc:330] stopping tablet replica
I20260929 18:33:40.740002 14874 raft_consensus.cc:2229] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:40.740329 14874 raft_consensus.cc:2258] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:40.741237 14874 ts_tablet_manager.cc:1905] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:40.744577 14874 ts_tablet_manager.cc:1918] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:40.744696 14874 log.cc:1196] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/03f2f77f333b45fe9cf9742cd5a8335a
I20260929 18:33:40.744975 14874 ts_tablet_manager.cc:1939] T 03f2f77f333b45fe9cf9742cd5a8335a P aec27c80404e45f48714fd33ad0726ca: Deleting consensus metadata
I20260929 18:33:40.746522 14660 catalog_manager.cc:4841] TS aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215): tablet 03f2f77f333b45fe9cf9742cd5a8335a (table test-table [id=c330c3728e114751bf702b4ad64997c0]) successfully deleted
W20260929 18:33:40.821941 14869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:40.822190 14869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:40.822232 14869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:40.826152 14869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:40.826203 14869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:40.826259 14869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:40.831403 14869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37483
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:40.832368 14869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:40.833287 14869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:40.842690 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:40.842700 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.842937 14869 server_base.cc:1034] running on GCE node
W20260929 18:33:40.842767 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:40.843511 14869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:40.844190 14869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:40.845362 14869 hybrid_clock.cc:648] HybridClock initialized: now 1790706820845353 us; error 31 us; skew 500 ppm
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:40.848141 14869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:40.849030 14869 webserver.cc:460] Webserver started at http://127.0.98.2:37483/ using document root <none> and password file <none>
I20260929 18:33:40.849606 14869 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:40.849681 14869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:40.853525 14869 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:33:40.855589 14889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.856923 14869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:40.857055 14869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "1b21885c85ac4a52a5c26425ad447dbe"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b1ffe6e0a871d18ff1bd322e18f84ddc"
server_key_iv: "d3923ab9fa89ede8a3f4e457e8726702"
server_key_version: "encryptionkey@0"
I20260929 18:33:40.857653 14869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:40.875073 14869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:40.878199 14869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:40.878413 14869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:40.879128 14869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:40.880118 14869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:40.880218 14869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.880280 14869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:40.880321 14869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:40.889866 14869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43061
I20260929 18:33:40.889887 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43061 every 8 connection(s)
I20260929 18:33:40.890902 14869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:40.900395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 14869
Sep 29 18:33:40 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:40.900884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40085
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=38371
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:40.905361 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.892821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55109 (local address 127.0.98.62:38557)
0929 18:33:40.892967 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:40.892971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:40.893963 (+ 992us) server_negotiation.cc:407] Connection header received
0929 18:33:40.894896 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:40.894900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:40.894963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:40.895063 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:40.896237 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.896767 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.897903 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.898151 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.898267 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.901245 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:40.901248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:40.901250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:40.901274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:40.903004 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.903521 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.903527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.903530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.903629 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:40.903992 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:40.903998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:40.904003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:40.904298 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:40.904404 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:40.904952 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 18:33:40.905109 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:33:40.906085 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:40.893193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:55109)
0929 18:33:40.893814 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:40.893855 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:40.894679 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:40.895241 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:40.895247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:40.895602 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:40.896063 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.896075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.896892 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.896898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.897777 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:40.897786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:40.898306 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:40.898310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:40.898579 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:40.899313 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:40.899339 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:40.901075 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:40.903127 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.903132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.903143 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.903391 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.903733 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:40.903736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:40.903738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:40.903895 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:40.904428 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:40.904433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:40.904699 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 18:33:40.904865 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:33:40.905198 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":101,"threads_started":1}
I20260929 18:33:40.907337 15003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:40.907624 15003 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:40.908098 15003 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:40.909227 14676 ts_manager.cc:194] Registered new tserver with Master: 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2:43061)
I20260929 18:33:40.909813 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55109
W20260929 18:33:41.013578 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:41.013831 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:41.013872 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:41.017911 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:41.017970 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:41.018064 15006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:41.023253 15006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38371
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:41.024271 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:41.025267 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:41.034772 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.034814 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.034793 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:41.034866 15006 server_base.cc:1034] running on GCE node
I20260929 18:33:41.035368 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:41.036003 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:41.037189 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790706821037171 us; error 28 us; skew 500 ppm
Sep 29 18:33:41 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:41.040300 15006 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:41.041316 15006 webserver.cc:460] Webserver started at http://127.0.98.3:38371/ using document root <none> and password file <none>
I20260929 18:33:41.041815 15006 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:41.041887 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:41.045681 15006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:41.047678 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.049042 15006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:41.049180 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "2c53d71b8ab04fd381075b330e4c4265"
format_stamp: "Formatted at 2026-09-29 18:33:40 on dist-test-slave-fmv8"
server_key: "de026c1d127c15c2f7dcad22cdda3cf4"
server_key_iv: "ef08d56f1054388d5f15b685fb834f82"
server_key_version: "encryptionkey@0"
I20260929 18:33:41.049693 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:41.077766 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:41.081318 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:41.081521 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:41.082258 15006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:41.083106 15006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:41.083156 15006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.083197 15006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:41.083256 15006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.092101 15006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40085
I20260929 18:33:41.092131 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40085 every 8 connection(s)
I20260929 18:33:41.093134 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:41.098138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15006
Sep 29 18:33:41 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:41.107220 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.098880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.0.98.62:38557)
0929 18:33:41.099193 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.099196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.099268 (+ 72us) server_negotiation.cc:407] Connection header received
0929 18:33:41.099403 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.099406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.099453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.099547 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:41.100218 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.100916 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.102070 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.102421 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.102527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.103987 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:41.103991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:41.103993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:41.104019 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:41.105548 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:41.106045 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:41.106047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:41.106049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:41.106117 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:41.106384 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:41.106387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:41.106389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:41.106565 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:41.106774 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.106912 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.107026 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260929 18:33:41.108191 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.095076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51871 (local address 127.0.98.62:38557)
0929 18:33:41.095235 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.095239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.096305 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 18:33:41.097193 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.097197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.097256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.097335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:41.098567 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.099278 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.100776 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.101022 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.101143 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.104083 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:41.104087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:41.104090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:41.104133 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:41.105735 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:41.106291 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:41.106293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:41.106294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:41.106341 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:41.106710 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:41.106715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:41.106737 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:41.106950 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:41.107052 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.107776 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.107871 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:33:41.109138 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.095456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:51871)
0929 18:33:41.096101 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.096199 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.096994 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.097496 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:41.097503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:41.097902 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:41.098413 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.098427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.099429 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.099433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.100647 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.100658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.101163 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.101166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.101298 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.101889 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:41.101905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:41.103895 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:41.105837 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:41.105843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:41.105861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:41.106209 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:41.106434 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:41.106439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:41.106441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:41.106580 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:41.107113 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:41.107121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:41.107458 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 18:33:41.107707 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 18:33:41.108066 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":114,"threads_started":1}
I20260929 18:33:41.110179 15135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 18:33:41.110428 15135 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:41.110908 15135 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 18:33:41.111919 14676 ts_manager.cc:194] Registered new tserver with Master: 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085)
I20260929 18:33:41.112505 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51871
I20260929 18:33:41.121435 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:41.125689 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.122022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.0.98.62:38557)
0929 18:33:41.122156 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.122160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.122311 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:33:41.122457 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.122459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.122553 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.122631 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:41.122809 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.123293 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.124207 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.124445 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.124535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.124930 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:41.125326 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:41.125427 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.125540 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.125588 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:33:41.126365 14676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:33:41.129789 14676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:33:41.137418 14799 tablet_service.cc:1467] Processing CreateTablet for tablet d0b97720c6634ec2b74108295e05c261 (DEFAULT_TABLE table=test-table [id=365130cf6efc4d75b52959877239ce30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:41.138064 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b97720c6634ec2b74108295e05c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:41.140982 14757 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Bootstrap starting.
I20260929 18:33:41.141773 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.136603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.0.1:55126)
0929 18:33:41.136980 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.136994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.137077 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.137868 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:41.137871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:41.137889 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:41.137975 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.137981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.139279 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.139283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.140316 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.140325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.141384 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.141389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.141483 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.141520 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:33:41.141563 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 18:33:41.141617 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":91,"threads_started":1}
I20260929 18:33:41.141865 14757 tablet_bootstrap.cc:654] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:41.142226 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.136371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43061 (local address 127.0.0.1:38306)
0929 18:33:41.136468 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.136484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.136563 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.137426 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:41.137428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:41.137446 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:41.137554 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.137561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.139453 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.139457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.140721 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.140733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.141797 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.141803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.141912 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.141938 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:33:41.141996 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:33:41.142071 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":67}
I20260929 18:33:41.142350 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.137034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.3:40085)
0929 18:33:41.137358 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.137370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.137391 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:33:41.137467 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.137472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.137683 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.137871 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:41.138075 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.139136 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.140446 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.141251 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.141970 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.142102 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.142143 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.142214 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":145,"threads_started":1}
I20260929 18:33:41.142923 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.136564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.98.2:43061)
0929 18:33:41.136904 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.136916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.136941 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:33:41.137025 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.137031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.137267 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.137456 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:41.137649 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.139315 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.140878 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.141643 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.142410 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.142563 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.142615 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.142737 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":125,"threads_started":1}
I20260929 18:33:41.143546 14757 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: No bootstrap required, opened a new log
I20260929 18:33:41.143643 14757 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:41.144037 14757 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.144131 14757 raft_consensus.cc:374] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:41.144183 14757 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec27c80404e45f48714fd33ad0726ca, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.144279 14757 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.144526 14757 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:33:41.145313 15069 tablet_service.cc:1467] Processing CreateTablet for tablet d0b97720c6634ec2b74108295e05c261 (DEFAULT_TABLE table=test-table [id=365130cf6efc4d75b52959877239ce30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:41.146737 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b97720c6634ec2b74108295e05c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:41.147221 14937 tablet_service.cc:1467] Processing CreateTablet for tablet d0b97720c6634ec2b74108295e05c261 (DEFAULT_TABLE table=test-table [id=365130cf6efc4d75b52959877239ce30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:41.148308 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b97720c6634ec2b74108295e05c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:41.153469 15148 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Bootstrap starting.
I20260929 18:33:41.155059 15149 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Bootstrap starting.
I20260929 18:33:41.155609 15148 tablet_bootstrap.cc:654] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:41.156893 15148 log.cc:824] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:41.157574 15149 tablet_bootstrap.cc:654] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:41.158243 15149 log.cc:824] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:41.158905 15148 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: No bootstrap required, opened a new log
I20260929 18:33:41.159111 15148 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:33:41.160238 15149 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: No bootstrap required, opened a new log
I20260929 18:33:41.160430 15149 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:33:41.162487 15148 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.162698 15148 raft_consensus.cc:374] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:41.162755 15148 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c53d71b8ab04fd381075b330e4c4265, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.163208 15148 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.163848 15135 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 18:33:41.163656 15149 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.163931 15149 raft_consensus.cc:374] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:41.163992 15149 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.164249 15148 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Time spent starting tablet: real 0.005s user 0.001s sys 0.006s
I20260929 18:33:41.164451 15149 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.165107 15003 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 18:33:41.165402 15149 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 18:33:41.200645 14866 tablet.cc:2369] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:33:41.274024 15153 raft_consensus.cc:482] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:41.274290 15153 raft_consensus.cc:504] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.275519 15153 leader_election.cc:288] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085), aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
I20260929 18:33:41.279570 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.275732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.98.2:51111)
0929 18:33:41.275918 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.275933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.276029 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.276376 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:41.276379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:41.276399 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:41.276522 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.276531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.277401 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.277404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.278312 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.278320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.279139 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.279143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.279251 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.279277 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:33:41.279355 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:33:41.279426 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":28}
I20260929 18:33:41.279599 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.275848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51111 (local address 127.0.98.3:40085)
0929 18:33:41.275993 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.275997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.276015 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:33:41.276180 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.276184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.276250 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.276329 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:41.276664 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.277294 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.278434 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.278980 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.279160 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.279221 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.279386 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.279471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:33:41.280084 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.275819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.98.2:48977)
0929 18:33:41.276197 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.276210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.276320 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.276662 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:41.276665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:41.276684 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:41.276805 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.276812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.277625 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.277627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.278653 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:41.278663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.279510 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.279513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:41.279770 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.279806 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:33:41.279888 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:33:41.279949 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":46,"thread_start_us":136,"threads_started":1}
I20260929 18:33:41.280164 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.275911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48977 (local address 127.0.98.1:40215)
0929 18:33:41.276228 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.276232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.276289 (+ 57us) server_negotiation.cc:407] Connection header received
0929 18:33:41.276417 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.276420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.276485 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.276605 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:41.276920 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.277453 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.278766 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.279404 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.279497 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.279659 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.279910 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.280000 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260929 18:33:41.280310 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c53d71b8ab04fd381075b330e4c4265" is_pre_election: true
I20260929 18:33:41.280712 15089 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 0.
I20260929 18:33:41.280982 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca" is_pre_election: true
I20260929 18:33:41.281226 14891 leader_election.cc:302] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, 2c53d71b8ab04fd381075b330e4c4265; no voters:
I20260929 18:33:41.281282 14819 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 0.
I20260929 18:33:41.281514 15153 raft_consensus.cc:2781] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:41.281625 15153 raft_consensus.cc:482] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:41.281692 15153 raft_consensus.cc:3037] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:41.282835 15153 raft_consensus.cc:504] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.283185 15153 leader_election.cc:288] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 election: Requested vote from peers 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085), aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
I20260929 18:33:41.283573 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca"
I20260929 18:33:41.283812 14819 raft_consensus.cc:3037] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:41.284904 14819 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 1.
I20260929 18:33:41.285156 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c53d71b8ab04fd381075b330e4c4265"
I20260929 18:33:41.285218 14890 leader_election.cc:302] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, aec27c80404e45f48714fd33ad0726ca; no voters:
I20260929 18:33:41.285274 15089 raft_consensus.cc:3037] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:41.285404 15153 raft_consensus.cc:2781] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:41.285643 15153 raft_consensus.cc:686] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 LEADER]: Becoming Leader. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Running, Role: LEADER
I20260929 18:33:41.285983 15153 consensus_queue.cc:237] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.286520 15089 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 1.
I20260929 18:33:41.289345 14676 catalog_manager.cc:5495] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b21885c85ac4a52a5c26425ad447dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } health_report { overall_health: UNKNOWN } } }
I20260929 18:33:41.303553 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.299871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.98.2:43061)
0929 18:33:41.300000 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:41.300004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:41.300017 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:33:41.300254 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:41.300256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:41.300307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:41.300397 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:41.300691 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.301183 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.302042 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:41.302277 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:41.302367 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:41.302707 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:41.303133 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:41.303248 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:41.303376 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:33:41.303429 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:33:41.309484 15089 raft_consensus.cc:1261] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Refusing update from remote peer 1b21885c85ac4a52a5c26425ad447dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:41.309633 14819 raft_consensus.cc:1261] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Refusing update from remote peer 1b21885c85ac4a52a5c26425ad447dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:41.310331 15153 consensus_queue.cc:1035] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:41.310537 15156 consensus_queue.cc:1035] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:41.317081 15159 mvcc.cc:205] Tried to move back new op lower bound from 7334735140076351488 to 7334735139990716416. Current Snapshot: MvccSnapshot[applied={T|T < 7334735140076351488}]
I20260929 18:33:41.318954 15160 mvcc.cc:205] Tried to move back new op lower bound from 7334735140076351488 to 7334735139990716416. Current Snapshot: MvccSnapshot[applied={T|T < 7334735140076351488}]
I20260929 18:33:41.325294 14675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:41.325457 14675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:41.327911 14675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Sending DeleteTablet for 3 replicas of tablet d0b97720c6634ec2b74108295e05c261
I20260929 18:33:41.328622 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b97720c6634ec2b74108295e05c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38306
I20260929 18:33:41.328788 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b97720c6634ec2b74108295e05c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55126
I20260929 18:33:41.328846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14714
I20260929 18:33:41.329022 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b97720c6634ec2b74108295e05c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41056
I20260929 18:33:41.329977 15168 tablet_replica.cc:330] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: stopping tablet replica
I20260929 18:33:41.330333 15168 raft_consensus.cc:2229] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:41.330639 15168 raft_consensus.cc:2258] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:41.332084 15168 ts_tablet_manager.cc:1905] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:41.335494 15168 ts_tablet_manager.cc:1918] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:41.335609 15168 log.cc:1196] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/d0b97720c6634ec2b74108295e05c261
I20260929 18:33:41.335942 15168 ts_tablet_manager.cc:1939] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Deleting consensus metadata
I20260929 18:33:41.336395 15167 tablet_replica.cc:330] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: stopping tablet replica
I20260929 18:33:41.336743 15167 raft_consensus.cc:2229] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:33:41.337112 15167 raft_consensus.cc:2258] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:41.337174 14661 catalog_manager.cc:4841] TS 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085): tablet d0b97720c6634ec2b74108295e05c261 (table test-table [id=365130cf6efc4d75b52959877239ce30]) successfully deleted
W20260929 18:33:41.337294 14660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40215 (error 108)
I20260929 18:33:41.338357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14869
I20260929 18:33:41.340314 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.339960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.0.1:41072)
0929 18:33:41.340088 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.340204 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 18:33:41.340729 14660 catalog_manager.cc:4584] TS aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0b97720c6634ec2b74108295e05c261: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40215: connect: Connection refused (error 111)
I20260929 18:33:41.344518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15006
I20260929 18:33:41.345492 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.344777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43061 (local address 127.0.0.1:38326)
0929 18:33:41.345308 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.345388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
I20260929 18:33:41.350446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 14643
I20260929 18:33:41.357005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--webserver_interface=127.0.98.62
--webserver_port=40483
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:41.466535 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:41.466858 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:41.466933 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:41.472333 15170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:41.472414 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:41.472450 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:41.472486 15170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:41.472520 15170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:41.477864 15170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:41.479014 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:41.480088 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:41.488466 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.488580 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.488704 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:41.488930 15170 server_base.cc:1034] running on GCE node
I20260929 18:33:41.489456 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:41.490440 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:41.491650 15170 hybrid_clock.cc:648] HybridClock initialized: now 1790706821491633 us; error 29 us; skew 500 ppm
Sep 29 18:33:41 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:41.494270 15170 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:41.495134 15170 webserver.cc:460] Webserver started at http://127.0.98.62:40483/ using document root <none> and password file <none>
I20260929 18:33:41.495626 15170 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:41.495683 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:41.499266 15170 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 18:33:41.501266 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.502436 15170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:33:41.502528 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4ec3f6ef90414ca7ae17097398fc294f"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b88cdf02b9c06374667393d9cda5970f"
server_key_iv: "a7c05421588f8b107b667c2b013c5baf"
server_key_version: "encryptionkey@0"
I20260929 18:33:41.503080 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:41.527621 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:41.530663 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:41.530861 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:41.537921 15170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38557
I20260929 18:33:41.538208 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38557 every 8 connection(s)
I20260929 18:33:41.539002 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:41.543203 15238 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:33:41.543371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15170
I20260929 18:33:41.543846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40215
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=43783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:41.545431 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap starting.
I20260929 18:33:41.562374 15238 log.cc:824] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:41.573249 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:41.573629 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f: Bootstrap complete.
I20260929 18:33:41.577551 15238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:41.577976 15238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.578451 15238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:41.578655 15238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:41.578725 15238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:41.578882 15238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:33:41.580905 15238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:41.581291 15238 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec3f6ef90414ca7ae17097398fc294f; no voters:
I20260929 18:33:41.581632 15238 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:33:41.581712 15242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:33:41.581895 15242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [term 3 LEADER]: Becoming Leader. State: Replica: 4ec3f6ef90414ca7ae17097398fc294f, State: Running, Role: LEADER
I20260929 18:33:41.582170 15242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 18:33:41.582795 15238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:41.583065 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec3f6ef90414ca7ae17097398fc294f. Latest consensus state: current_term: 3 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:41.583288 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:41.583606 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4ec3f6ef90414ca7ae17097398fc294f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec3f6ef90414ca7ae17097398fc294f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 18:33:41.583779 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec3f6ef90414ca7ae17097398fc294f [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:41.583665 15246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:41.589118 15246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:41.589838 15246 catalog_manager.cc:1259] Loaded cluster ID: ab9b646acc414956ada398233b6efe67
I20260929 18:33:41.589900 15246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:33:41.590862 15246 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fedbbf7a9ad google::LogMessage::Fail() at ??:0
@ 0x7fedbbf7d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fedbbf7a4de google::LogMessage::Flush() at ??:0
@ 0x7fedbbf7ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fedc68b9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fedc68bad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fedc68b7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fedc68fb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fedbe1cc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fedbe1ccc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fedbe1ce3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fedbe1bd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fedbf8016db start_thread at ??:0
@ 0x7fedbaa7861f clone at ??:0
W20260929 18:33:41.675791 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:41.676066 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:41.676137 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:41.680522 15240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:41.680598 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:41.680701 15240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:41.686470 15240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43783
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:41.687498 15240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:41.688565 15240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:41.698104 15279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.698119 15280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.698200 15282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:41.698477 15240 server_base.cc:1034] running on GCE node
I20260929 18:33:41.699071 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:41.699656 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:41.700858 15240 hybrid_clock.cc:648] HybridClock initialized: now 1790706821700832 us; error 42 us; skew 500 ppm
Sep 29 18:33:41 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:41.703372 15240 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:41.704355 15240 webserver.cc:460] Webserver started at http://127.0.98.1:43783/ using document root <none> and password file <none>
I20260929 18:33:41.704984 15240 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:41.705057 15240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:41.708810 15240 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:33:41.711225 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.712595 15240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:33:41.712738 15240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "aec27c80404e45f48714fd33ad0726ca"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "13a11e7cdda411f85e6200b8a3f996cb"
server_key_iv: "ed326adb72363b86c6051064cf8e344f"
server_key_version: "encryptionkey@0"
I20260929 18:33:41.713286 15240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:41.732007 15240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:41.735229 15240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:41.735431 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:41.735993 15240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:41.737113 15296 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:33:41.742471 15240 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:33:41.742544 15240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 18:33:41.742646 15240 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:33:41.744256 15240 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:33:41.744318 15240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:41.744604 15296 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Bootstrap starting.
I20260929 18:33:41.754804 15240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40215
I20260929 18:33:41.755064 15403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40215 every 8 connection(s)
I20260929 18:33:41.755807 15240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:41.760802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15240
I20260929 18:33:41.761209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43061
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=37483
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:41.765378 15296 log.cc:824] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:41.774911 15296 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:41.775311 15296 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Bootstrap complete.
I20260929 18:33:41.776773 15296 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.016s
I20260929 18:33:41.779857 15296 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.780392 15296 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec27c80404e45f48714fd33ad0726ca, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.780941 15296 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.781984 15296 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 18:33:41.815480 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:41.757684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:58255)
0929 18:33:41.758150 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.758203 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:41.759018 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:41.814737 (+ 55719us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":147,"threads_started":1}
W20260929 18:33:41.816083 15404 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38557: Client connection negotiation failed: client connection to 127.0.98.62:38557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:33:41.816886 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.816609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:59583)
0929 18:33:41.816742 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.816820 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 18:33:41.817602 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.817395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:55023)
0929 18:33:41.817503 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.817552 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:33:41.817795 15404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:33:41.877684 15407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:41.877970 15407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:41.878049 15407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:41.882324 15407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:41.882404 15407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:41.882489 15407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:41.887782 15407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37483
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:41.888886 15407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:41.889887 15407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:41.899573 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:41.899554 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:41.899607 15407 server_base.cc:1034] running on GCE node
W20260929 18:33:41.899546 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:41.900225 15407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:41.900848 15407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:41.902024 15407 hybrid_clock.cc:648] HybridClock initialized: now 1790706821901994 us; error 50 us; skew 500 ppm
Sep 29 18:33:41 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:41.904790 15407 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:41.905751 15407 webserver.cc:460] Webserver started at http://127.0.98.2:37483/ using document root <none> and password file <none>
I20260929 18:33:41.906304 15407 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:41.906392 15407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:41.910112 15407 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:33:41.912217 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:41.913371 15407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 18:33:41.913511 15407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "1b21885c85ac4a52a5c26425ad447dbe"
format_stamp: "Formatted at 2026-09-29 18:33:39 on dist-test-slave-fmv8"
server_key: "b1ffe6e0a871d18ff1bd322e18f84ddc"
server_key_iv: "d3923ab9fa89ede8a3f4e457e8726702"
server_key_version: "encryptionkey@0"
I20260929 18:33:41.914041 15407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:41.930943 15407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:41.934298 15407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:41.934507 15407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:41.935135 15407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:41.936256 15431 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:33:41.940205 15407 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:33:41.940284 15407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:33:41.940362 15407 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:33:41.941978 15407 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:33:41.942052 15407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:33:41.942296 15431 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Bootstrap starting.
I20260929 18:33:41.947293 15431 log.cc:824] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:41.953003 15407 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43061
I20260929 18:33:41.953029 15539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43061 every 8 connection(s)
I20260929 18:33:41.953823 15407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:41.955055 15431 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:41.955437 15431 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Bootstrap complete.
I20260929 18:33:41.955746 15431 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260929 18:33:41.957160 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.955677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:35397)
0929 18:33:41.956202 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.956329 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
W20260929 18:33:41.957647 15540 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38557: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
I20260929 18:33:41.958312 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.958070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:32905)
0929 18:33:41.958204 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.958253 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:33:41.958914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15407
I20260929 18:33:41.959034 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.958806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:51749)
0929 18:33:41.958917 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.958973 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:33:41.959233 15540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:33:41.959411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40085
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38557
--webserver_port=38371
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:33:41.959412 15431 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.959918 15431 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Initialized, Role: FOLLOWER
I20260929 18:33:41.960384 15431 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:41.961287 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.961067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:47589)
0929 18:33:41.961190 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.961233 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 18:33:41.961341 15431 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 18:33:41.961998 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:41.961775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:37477)
0929 18:33:41.961872 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:41.961927 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:33:42.067862 15543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:42.068109 15543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:42.068246 15543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:42.072099 15543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:42.072214 15543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:42.072297 15543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:42.077399 15543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38371
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:42.078375 15543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:42.079463 15543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:42.089378 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:42.089434 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:42.089412 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:42.089597 15543 server_base.cc:1034] running on GCE node
I20260929 18:33:42.089974 15543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:42.090577 15543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:42.091754 15543 hybrid_clock.cc:648] HybridClock initialized: now 1790706822091721 us; error 45 us; skew 500 ppm
Sep 29 18:33:42 dist-test-slave-fmv8 krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:42.094624 15543 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:42.095522 15543 webserver.cc:460] Webserver started at http://127.0.98.3:38371/ using document root <none> and password file <none>
I20260929 18:33:42.096186 15543 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:42.096262 15543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:42.099957 15543 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:42.102044 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:42.103282 15543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:42.103412 15543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "2c53d71b8ab04fd381075b330e4c4265"
format_stamp: "Formatted at 2026-09-29 18:33:40 on dist-test-slave-fmv8"
server_key: "de026c1d127c15c2f7dcad22cdda3cf4"
server_key_iv: "ef08d56f1054388d5f15b685fb834f82"
server_key_version: "encryptionkey@0"
I20260929 18:33:42.103941 15543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:42.127120 15543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:42.130141 15543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:42.130347 15543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:42.130934 15543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:42.131750 15543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:42.131821 15543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:42.131893 15543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:42.131937 15543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:42.140465 15543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40085
I20260929 18:33:42.140496 15672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40085 every 8 connection(s)
I20260929 18:33:42.141413 15543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:42.145207 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:42.143492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:46505)
0929 18:33:42.144422 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:42.144524 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":543,"threads_started":1}
W20260929 18:33:42.145624 15673 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38557: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
I20260929 18:33:42.146087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15543
I20260929 18:33:42.146220 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:42.145973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:48429)
0929 18:33:42.146081 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:42.146138 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 18:33:42.146970 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:42.146769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:49607)
0929 18:33:42.146872 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:42.146916 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:33:42.147116 15673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:33:42.818863 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:42.818307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:52233)
0929 18:33:42.818624 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:42.818713 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":141,"threads_started":1}
I20260929 18:33:42.962947 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:42.962597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:53463)
0929 18:33:42.962816 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:42.962868 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260929 18:33:43.148072 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.147663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:37719)
0929 18:33:43.147917 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.147973 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260929 18:33:43.245873 15680 raft_consensus.cc:482] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:43.246085 15680 raft_consensus.cc:504] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:43.247268 15680 leader_election.cc:288] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085), aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
Sep 29 18:33:43 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 18:33:43 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:33:43.263535 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.247742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41981 (local address 127.0.98.1:40215)
0929 18:33:43.248045 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:43.248052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:43.248082 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:33:43.248604 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:43.248609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:43.248811 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:43.249003 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:43.250640 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.252205 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.254119 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.254888 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.255059 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:43.258615 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:43.258622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:43.258638 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:43.258670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:43.261029 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261632 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:43.261637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:43.261641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:43.261761 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:43.262106 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:43.262113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:43.262115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:43.262483 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:43.262628 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:43.263099 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 18:33:43.263350 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260929 18:33:43.263538 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.247633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32947 (local address 127.0.98.3:40085)
0929 18:33:43.248177 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:43.248200 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:43.248245 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:33:43.249073 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:43.249079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:43.249355 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:43.249666 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:43.250641 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.252205 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.254119 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.254866 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.255046 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:43.258675 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:43.258680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:43.258690 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:43.258720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:43.261029 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261594 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:43.261602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:43.261609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:43.261727 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:43.262082 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:43.262088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:43.262092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:43.262471 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:43.262603 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:43.263112 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 18:33:43.263348 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":72,"threads_started":1}
I20260929 18:33:43.263593 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.247559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40215 (local address 127.0.98.2:41981)
0929 18:33:43.247884 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.247899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:43.248260 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:43.249105 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:43.249115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:43.249641 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:43.250426 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:43.250468 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.252369 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.252373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:43.253973 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:43.253984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.255012 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.255015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:43.255309 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:43.256102 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:43.256130 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:43.258447 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:43.261139 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:43.261155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:43.261488 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:43.261878 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:43.261881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:43.261986 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:43.262610 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:43.262615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:43.262809 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 18:33:43.262993 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:33:43.263338 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":294,"spinlock_wait_cycles":63872,"thread_start_us":82,"threads_started":1}
I20260929 18:33:43.264840 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.247429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.98.2:32947)
0929 18:33:43.247697 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.247730 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:43.248250 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:43.249623 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:43.249626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:43.249791 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:43.250426 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:43.250468 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.252369 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.252373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:43.253973 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:43.253984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:43.255004 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:43.255008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:43.255311 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:43.256102 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:43.256130 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:43.258040 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:43.261191 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:43.261206 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:43.261480 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:43.261824 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:43.261828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:43.261830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:43.261984 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:43.262591 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:43.262597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:43.262810 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 18:33:43.262994 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:33:43.263432 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":480,"spinlock_wait_cycles":23680}
I20260929 18:33:43.264810 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca" is_pre_election: true
I20260929 18:33:43.265157 15358 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 1.
I20260929 18:33:43.266177 15425 leader_election.cc:302] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, aec27c80404e45f48714fd33ad0726ca; no voters:
I20260929 18:33:43.266469 15680 raft_consensus.cc:2781] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:33:43.266542 15680 raft_consensus.cc:482] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:43.266572 15680 raft_consensus.cc:3037] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 1 FOLLOWER]: Advancing to term 2
W20260929 18:33:43.267073 15426 leader_election.cc:341] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085): Not found: Tablet not found: d0b97720c6634ec2b74108295e05c261
I20260929 18:33:43.268746 15680 raft_consensus.cc:504] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:43.269102 15680 leader_election.cc:288] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 election: Requested vote from peers 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085), aec27c80404e45f48714fd33ad0726ca (127.0.98.1:40215)
I20260929 18:33:43.269568 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b97720c6634ec2b74108295e05c261" candidate_uuid: "1b21885c85ac4a52a5c26425ad447dbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "aec27c80404e45f48714fd33ad0726ca"
I20260929 18:33:43.269776 15358 raft_consensus.cc:3037] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 1 FOLLOWER]: Advancing to term 2
W20260929 18:33:43.269964 15426 leader_election.cc:341] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085): Not found: Tablet not found: d0b97720c6634ec2b74108295e05c261
I20260929 18:33:43.272291 15358 raft_consensus.cc:2454] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b21885c85ac4a52a5c26425ad447dbe in term 2.
I20260929 18:33:43.272593 15425 leader_election.cc:302] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b21885c85ac4a52a5c26425ad447dbe, aec27c80404e45f48714fd33ad0726ca; no voters: 2c53d71b8ab04fd381075b330e4c4265
I20260929 18:33:43.272827 15680 raft_consensus.cc:2781] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:33:43.273078 15680 raft_consensus.cc:686] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [term 2 LEADER]: Becoming Leader. State: Replica: 1b21885c85ac4a52a5c26425ad447dbe, State: Running, Role: LEADER
I20260929 18:33:43.273414 15680 consensus_queue.cc:237] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:43.273984 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.273646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:49011)
0929 18:33:43.273826 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.273904 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 18:33:43.810330 15426 consensus_peers.cc:487] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe -> Peer 2c53d71b8ab04fd381075b330e4c4265 (127.0.98.3:40085): Couldn't send request to peer 2c53d71b8ab04fd381075b330e4c4265. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0b97720c6634ec2b74108295e05c261. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:33:43.819979 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.819572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:48153)
0929 18:33:43.819828 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.819884 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260929 18:33:43.881227 15358 raft_consensus.cc:1261] T d0b97720c6634ec2b74108295e05c261 P aec27c80404e45f48714fd33ad0726ca [term 2 FOLLOWER]: Refusing update from remote peer 1b21885c85ac4a52a5c26425ad447dbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 18:33:43.881976 15686 consensus_queue.cc:1035] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 18:33:43.884111 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.883831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:51201)
0929 18:33:43.883960 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.884024 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:33:43.886858 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.886636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:38883)
0929 18:33:43.886753 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.886801 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:33:43.886858 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:43.886313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:38067)
0929 18:33:43.886695 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:43.886753 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":51,"threads_started":1}
I20260929 18:33:44.149190 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.148761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:53961)
0929 18:33:44.149043 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.149092 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260929 18:33:44.299984 15695 ts_tablet_manager.cc:927] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Initiating tablet copy from peer 1b21885c85ac4a52a5c26425ad447dbe (127.0.98.2:43061)
I20260929 18:33:44.300772 15695 tablet_copy_client.cc:316] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:43061
Sep 29 18:33:44 dist-test-slave-fmv8 krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 18:33:44.310940 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.301302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43061 (local address 127.0.98.3:57649)
0929 18:33:44.301441 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.301458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:44.301566 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:44.302217 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:44.302223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:44.302618 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:44.302727 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:44.302733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:44.303711 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:44.303717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:44.304423 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:44.304433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:44.305024 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:44.305028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:44.305143 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:44.305663 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:44.305675 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:44.307083 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:44.309160 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:44.309171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:44.309178 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:44.309419 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:44.309727 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:44.309731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:44.309733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:44.309826 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:44.310280 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:44.310286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:44.310424 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:33:44.310571 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 18:33:44.310747 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:33:44.310957 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.301494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57649 (local address 127.0.98.2:43061)
0929 18:33:44.301806 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:44.301815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:44.301849 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:33:44.301891 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:44.301895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:44.302068 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:44.302219 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:44.302858 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:44.303599 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:44.304540 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:44.304905 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:44.305026 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:44.307227 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:44.307235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:44.307247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:44.307269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:44.309051 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:44.309547 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:44.309552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:44.309553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:44.309634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:44.309917 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:44.309922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:44.309923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:44.310164 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:44.310309 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:44.310656 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:33:44.310796 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260929 18:33:44.312072 15514 tablet_copy_service.cc:140] P 1b21885c85ac4a52a5c26425ad447dbe: Received BeginTabletCopySession request for tablet d0b97720c6634ec2b74108295e05c261 from peer 2c53d71b8ab04fd381075b330e4c4265 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57649)
I20260929 18:33:44.312261 15514 tablet_copy_service.cc:161] P 1b21885c85ac4a52a5c26425ad447dbe: Beginning new tablet copy session on tablet d0b97720c6634ec2b74108295e05c261 from peer 2c53d71b8ab04fd381075b330e4c4265 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57649: session id = 2c53d71b8ab04fd381075b330e4c4265-d0b97720c6634ec2b74108295e05c261
I20260929 18:33:44.313392 15514 tablet_copy_source_session.cc:205] T d0b97720c6634ec2b74108295e05c261 P 1b21885c85ac4a52a5c26425ad447dbe: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:33:44.317627 15695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b97720c6634ec2b74108295e05c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:44.324024 15695 tablet_copy_client.cc:791] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: tablet copy: Starting download of 0 data blocks...
I20260929 18:33:44.324235 15695 tablet_copy_client.cc:655] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: tablet copy: Starting download of 1 WAL segments...
I20260929 18:33:44.325979 15695 tablet_copy_client.cc:531] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:33:44.327548 15695 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Bootstrap starting.
I20260929 18:33:44.332790 15695 log.cc:824] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:44.339017 15695 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:33:44.339305 15695 tablet_bootstrap.cc:492] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Bootstrap complete.
I20260929 18:33:44.339613 15695 ts_tablet_manager.cc:1397] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 18:33:44.341821 15695 raft_consensus.cc:348] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:44.342069 15695 raft_consensus.cc:729] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c53d71b8ab04fd381075b330e4c4265, State: Initialized, Role: FOLLOWER
I20260929 18:33:44.342509 15695 consensus_queue.cc:260] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b21885c85ac4a52a5c26425ad447dbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43061 } } peers { permanent_uuid: "2c53d71b8ab04fd381075b330e4c4265" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "aec27c80404e45f48714fd33ad0726ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40215 } }
I20260929 18:33:44.343307 15695 ts_tablet_manager.cc:1428] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 18:33:44.343796 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.343511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:44697)
0929 18:33:44.343624 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.343689 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:33:44.344080 15514 tablet_copy_service.cc:342] P 1b21885c85ac4a52a5c26425ad447dbe: Request end of tablet copy session 2c53d71b8ab04fd381075b330e4c4265-d0b97720c6634ec2b74108295e05c261 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57649
I20260929 18:33:44.344219 15514 tablet_copy_service.cc:433] P 1b21885c85ac4a52a5c26425ad447dbe: ending tablet copy session 2c53d71b8ab04fd381075b330e4c4265-d0b97720c6634ec2b74108295e05c261 on tablet d0b97720c6634ec2b74108295e05c261 with peer 2c53d71b8ab04fd381075b330e4c4265
I20260929 18:33:44.344775 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.344550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:39321)
0929 18:33:44.344657 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.344702 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:33:44.720214 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.719788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:39211)
0929 18:33:44.720016 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.720062 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260929 18:33:44.720203 15627 raft_consensus.cc:1203] T d0b97720c6634ec2b74108295e05c261 P 2c53d71b8ab04fd381075b330e4c4265 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 18:33:44.721094 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.720888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:53641)
0929 18:33:44.720989 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.721035 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:33:44.887859 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:44.887494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:42531)
0929 18:33:44.887701 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:44.887754 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260929 18:33:45.721293 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:45.720905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:59917)
0929 18:33:45.721141 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:45.721201 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260929 18:33:45.722193 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:45.721779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:43873)
0929 18:33:45.722056 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:45.722118 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":73,"threads_started":1}
I20260929 18:33:45.889284 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:45.888753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:42399)
0929 18:33:45.889047 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:45.889112 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260929 18:33:46.722390 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:46.721995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:59985)
0929 18:33:46.722239 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:46.722296 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260929 18:33:46.723289 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:46.722957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:57367)
0929 18:33:46.723165 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:46.723215 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260929 18:33:46.890656 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:46.890206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:52691)
0929 18:33:46.890487 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:46.890543 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260929 18:33:47.723608 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:47.723208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:50541)
0929 18:33:47.723447 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:47.723513 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260929 18:33:47.724328 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:47.724080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:35359)
0929 18:33:47.724217 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:47.724268 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:33:47.724578 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:47.724129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:56289)
0929 18:33:47.724391 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:47.724474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260929 18:33:47.891986 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:47.891546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:59497)
0929 18:33:47.891823 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:47.891888 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260929 18:33:48.725566 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:48.725038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:60301)
0929 18:33:48.725328 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:48.725392 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260929 18:33:48.725770 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:48.725265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:60975)
0929 18:33:48.725577 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:48.725660 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260929 18:33:48.893200 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:48.892769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:53239)
0929 18:33:48.893048 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:48.893105 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260929 18:33:49.726771 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:49.726339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:39885)
0929 18:33:49.726615 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:49.726671 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260929 18:33:49.726836 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:49.726457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:50483)
0929 18:33:49.726697 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:49.726749 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260929 18:33:49.894474 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:49.894034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:36305)
0929 18:33:49.894314 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:49.894377 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260929 18:33:50.727952 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:50.727554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:60985)
0929 18:33:50.727808 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:50.727862 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260929 18:33:50.728025 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:50.727637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:39651)
0929 18:33:50.727874 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:50.727922 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260929 18:33:50.895833 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:50.895377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:57275)
0929 18:33:50.895666 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:50.895729 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260929 18:33:51.729249 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:51.728675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:53401)
0929 18:33:51.729109 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:51.729166 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
I20260929 18:33:51.729234 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:51.728773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:49189)
0929 18:33:51.729020 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:51.729118 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260929 18:33:51.897217 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:51.896772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:59451)
0929 18:33:51.897048 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:51.897116 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260929 18:33:52.730520 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:52.730086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:50537)
0929 18:33:52.730363 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:52.730428 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260929 18:33:52.730585 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:52.730115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:53205)
0929 18:33:52.730415 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:52.730467 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
I20260929 18:33:52.898468 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:52.898040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:57869)
0929 18:33:52.898312 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:52.898369 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260929 18:33:53.731709 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:53.731298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:37515)
0929 18:33:53.731561 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:53.731618 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":108,"threads_started":1}
I20260929 18:33:53.731770 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:53.731383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:52851)
0929 18:33:53.731632 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:53.731689 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260929 18:33:53.899749 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:53.899298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:47825)
0929 18:33:53.899596 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:53.899655 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
I20260929 18:33:54.732993 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:54.732517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:44017)
0929 18:33:54.732770 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:54.732834 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260929 18:33:54.733253 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:54.732647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:52487)
0929 18:33:54.732914 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:54.732982 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260929 18:33:54.901014 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:54.900602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:51957)
0929 18:33:54.900856 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:54.900917 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260929 18:33:55.734146 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:55.733738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:44749)
0929 18:33:55.733995 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:55.734053 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260929 18:33:55.734362 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:55.733939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:52621)
0929 18:33:55.734176 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:55.734234 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260929 18:33:55.902355 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:55.901870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:42289)
0929 18:33:55.902160 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:55.902222 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260929 18:33:56.735450 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:56.735031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:57111)
0929 18:33:56.735295 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:56.735355 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260929 18:33:56.735515 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:56.735031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:52565)
0929 18:33:56.735381 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:56.735431 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260929 18:33:56.903728 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:56.903319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:35669)
0929 18:33:56.903575 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:56.903630 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260929 18:33:57.147934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15240
I20260929 18:33:57.153822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15407
I20260929 18:33:57.159350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15543
2026-09-29T18:33:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19151 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:57 dist-test-slave-fmv8 krb5kdc[15742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:57 dist-test-slave-fmv8 krb5kdc[15742](info): set up 2 sockets
Sep 29 18:33:57 dist-test-slave-fmv8 krb5kdc[15742](info): commencing operation
krb5kdc: starting...
W20260929 18:33:58.192349 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:33:58.311141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:58.420372 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:58.420647 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:58.420727 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:58.424693 15755 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:33:58.424762 15755 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:33:58.424788 15755 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:33:58.424826 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:58.424870 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:58.424929 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:58.424968 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:58.425002 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:58.430145 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:58.431143 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:58.432106 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:58.440469 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:58.440544 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.440637 15755 server_base.cc:1034] running on GCE node
W20260929 18:33:58.440469 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.441437 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:58.442443 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:58.443607 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790706838443595 us; error 25 us; skew 500 ppm
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:58.446276 15755 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:33:58.447157 15755 webserver.cc:460] Webserver started at http://127.0.98.62:37135/ using document root <none> and password file <none>
I20260929 18:33:58.447647 15755 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:58.447702 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:58.447853 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:58.449807 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "0d8c403b5ec4420196cb1334e445a2c9"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "22122886a331716acd058db595c454ef"
server_key_iv: "388acbc60546eeab59c9d8dfbbbfed76"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.450278 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d8c403b5ec4420196cb1334e445a2c9"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "22122886a331716acd058db595c454ef"
server_key_iv: "388acbc60546eeab59c9d8dfbbbfed76"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.453773 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:33:58.455989 15770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.457237 15755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:33:58.457378 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0d8c403b5ec4420196cb1334e445a2c9"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "22122886a331716acd058db595c454ef"
server_key_iv: "388acbc60546eeab59c9d8dfbbbfed76"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.457487 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:58.515149 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:58.518321 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:58.518522 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:58.525575 15755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40839
I20260929 18:33:58.525856 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40839 every 8 connection(s)
I20260929 18:33:58.526520 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:58.527611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15755
I20260929 18:33:58.527701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:33:58.527990 392 external_mini_cluster.cc:1428] Setting key 083802ac891b5b40e72fa79fbfee7ec5
I20260929 18:33:58.529593 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:58.536367 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap starting.
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:58.538982 15823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:58.539886 15823 log.cc:824] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:58.542081 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: No bootstrap required, opened a new log
I20260929 18:33:58.543246 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.529333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.62:40839)
0929 18:33:58.529729 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:58.529743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:58.529779 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:33:58.530620 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:58.530646 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:58.531077 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:58.531466 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:58.532325 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.533920 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.535122 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.535570 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.535682 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:58.538353 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:58.538359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:58.538374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:58.538403 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:58.540380 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.540859 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.540865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.540870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.540992 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.541236 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.541239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.541240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.541719 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:58.541911 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:58.542081 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:33:58.542361 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260929 18:33:58.545652 15823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:33:58.545851 15823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:58.545884 15823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Initialized, Role: FOLLOWER
I20260929 18:33:58.546280 15823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:33:58.546406 15823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:58.546487 15823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:58.546581 15823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:58.547410 15823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:33:58.547730 15823 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8c403b5ec4420196cb1334e445a2c9; no voters:
I20260929 18:33:58.548048 15823 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:58.548144 15828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:58.548338 15828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Running, Role: LEADER
I20260929 18:33:58.548674 15828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:33:58.549273 15823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:58.550246 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8c403b5ec4420196cb1334e445a2c9. Latest consensus state: current_term: 1 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:33:58.550379 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:58.550756 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:33:58.551050 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:58.551435 15837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:58.554147 15837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:58.559095 15837 catalog_manager.cc:1347] Generated new cluster ID: c2bb05a427784640995c2950bd8984be
I20260929 18:33:58.559162 15837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:58.564651 15837 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:58.565768 15837 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:58.575275 15837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Generated new TSK 0
I20260929 18:33:58.576200 15837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:33:58.607290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:58.715323 15849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:58.715553 15849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:58.715593 15849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:58.719511 15849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:58.719565 15849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:58.719620 15849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:33:58.724596 15849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:58.725450 15849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:58.726331 15849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:58.735332 15854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:58.735376 15855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.735445 15849 server_base.cc:1034] running on GCE node
W20260929 18:33:58.735374 15857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.735941 15849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:58.736579 15849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:58.737782 15849 hybrid_clock.cc:648] HybridClock initialized: now 1790706838737770 us; error 22 us; skew 500 ppm
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:58.740489 15849 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:33:58.741253 15849 webserver.cc:460] Webserver started at http://127.0.98.1:35879/ using document root <none> and password file <none>
I20260929 18:33:58.741729 15849 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:58.741784 15849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:58.741941 15849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:58.743779 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "99dbc565a91342b1b1356f075a22d654"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "c8921f5866fefdd3614b492a64efb61c"
server_key_iv: "5ec170568fac390bc65601f1aa182ba7"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.744328 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "99dbc565a91342b1b1356f075a22d654"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "c8921f5866fefdd3614b492a64efb61c"
server_key_iv: "5ec170568fac390bc65601f1aa182ba7"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.747653 15849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:33:58.750012 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.751185 15849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:33:58.751271 15849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "99dbc565a91342b1b1356f075a22d654"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "c8921f5866fefdd3614b492a64efb61c"
server_key_iv: "5ec170568fac390bc65601f1aa182ba7"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.751348 15849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:58.764668 15849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:58.767632 15849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:58.767768 15849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:58.768301 15849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:58.768999 15849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:58.769052 15849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.769090 15849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:58.769105 15849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.777850 15849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260929 18:33:58.777892 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260929 18:33:58.778710 15849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:58.783674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15849
I20260929 18:33:58.783798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:58.784101 392 external_mini_cluster.cc:1428] Setting key e2b835724cd4d7f94b6163004ec59c36
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:58.793421 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.780535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48351 (local address 127.0.98.62:40839)
0929 18:33:58.780669 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:58.780673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:58.781459 (+ 786us) server_negotiation.cc:407] Connection header received
0929 18:33:58.782712 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:58.782716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:58.782769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:58.782854 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:58.784335 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.784887 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.786196 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.786636 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.786756 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:58.789274 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:58.789277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:58.789279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:58.789302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:58.790855 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.791480 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.791483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.791484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.791556 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.791958 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.791988 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.791989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.792202 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:58.792287 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:58.793194 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 18:33:58.793304 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:33:58.794543 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.780883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:48351)
0929 18:33:58.781310 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:58.781369 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:58.782512 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:58.783024 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:58.783030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:58.783446 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:58.784107 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:58.784128 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.785013 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.785017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:58.786059 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:58.786068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.786782 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.786785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:58.786944 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:58.787539 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:58.787555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:58.789111 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:58.790977 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:58.790984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:58.791000 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:58.791375 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:58.791654 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:58.791660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:58.791664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:58.791853 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:58.792310 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:58.792316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:58.792900 (+ 584us) client_negotiation.cc:769] Sending connection context
0929 18:33:58.793130 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 18:33:58.793493 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:33:58.795897 15978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:33:58.796185 15978 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:58.796765 15978 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:33:58.798337 15787 ts_manager.cc:194] Registered new tserver with Master: 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:33:58.799435 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:33:58.813184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:58.920478 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:58.920725 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:58.920799 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:58.924800 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:58.924853 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:58.924911 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:33:58.930178 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:58.931190 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:58.932307 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:58.941753 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:58.941812 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.941849 15983 server_base.cc:1034] running on GCE node
W20260929 18:33:58.941777 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.942391 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:58.943005 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:58.944219 15983 hybrid_clock.cc:648] HybridClock initialized: now 1790706838944204 us; error 25 us; skew 500 ppm
Sep 29 18:33:58 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:58.946844 15983 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:33:58.947721 15983 webserver.cc:460] Webserver started at http://127.0.98.2:43767/ using document root <none> and password file <none>
I20260929 18:33:58.948366 15983 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:58.948474 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:58.948668 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:58.950734 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "986d1c9a9da34f109edd050299afdae5"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "640a85a3e6e74067d8c08497bb2e41b9"
server_key_iv: "0f13f242bcede8654c19e3ec526f3a8b"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.951223 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "986d1c9a9da34f109edd050299afdae5"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "640a85a3e6e74067d8c08497bb2e41b9"
server_key_iv: "0f13f242bcede8654c19e3ec526f3a8b"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.954891 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 18:33:58.957334 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.958487 15983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:33:58.958593 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "986d1c9a9da34f109edd050299afdae5"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "640a85a3e6e74067d8c08497bb2e41b9"
server_key_iv: "0f13f242bcede8654c19e3ec526f3a8b"
server_key_version: "encryptionkey@0"
I20260929 18:33:58.958674 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:58.979022 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:58.982477 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:58.982646 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:58.983196 15983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:58.984048 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:58.984097 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.984136 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:58.984244 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.993315 15983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38457
I20260929 18:33:58.993341 16111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38457 every 8 connection(s)
I20260929 18:33:58.994238 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:58.998915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 15983
I20260929 18:33:58.999065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:58.999339 392 external_mini_cluster.cc:1428] Setting key 4e20af89cccd6a4df2eaaebd91046b93
Sep 29 18:33:59 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:59.009428 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.996104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49331 (local address 127.0.98.62:40839)
0929 18:33:58.996299 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:58.996303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:58.997145 (+ 842us) server_negotiation.cc:407] Connection header received
0929 18:33:58.998421 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:58.998425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:58.998481 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:58.998603 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:58.999958 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.000631 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.001840 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.002250 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.002370 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.005378 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:59.005383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:59.005387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:59.005421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:59.007125 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.007791 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.007793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.007795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.007868 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.008306 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.008309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.008311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.008497 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:59.008582 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.009215 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.009312 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 18:33:59.010576 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.996526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:49331)
0929 18:33:58.996967 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:58.997008 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:58.998208 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:58.998778 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:58.998786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:58.999236 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:58.999797 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:58.999816 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.000759 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.000764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.001710 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.001719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.002439 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.002444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.002698 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.003367 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:59.003389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:59.005195 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:59.007254 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.007264 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.007281 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.007685 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.007962 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.007965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.007967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.008189 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.008616 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:59.008623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:59.008950 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.009164 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.009565 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260929 18:33:59.011744 16112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:33:59.012005 16112 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:33:59.012522 16112 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
I20260929 18:33:59.013618 15787 ts_manager.cc:194] Registered new tserver with Master: 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
I20260929 18:33:59.014391 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:33:59.027673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:59.135684 16117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:59.135952 16117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:59.136021 16117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:59.140256 16117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:59.140329 16117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:59.140421 16117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:33:59.145475 16117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:33:59.146507 16117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:59.147476 16117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:59.157502 16122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:59.157533 16125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:59.157593 16117 server_base.cc:1034] running on GCE node
W20260929 18:33:59.157502 16123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:59.158164 16117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:59.158776 16117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:59.159976 16117 hybrid_clock.cc:648] HybridClock initialized: now 1790706839159946 us; error 40 us; skew 500 ppm
Sep 29 18:33:59 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:59.162851 16117 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:33:59.163699 16117 webserver.cc:460] Webserver started at http://127.0.98.3:35861/ using document root <none> and password file <none>
I20260929 18:33:59.164304 16117 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:59.164422 16117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:59.164647 16117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:59.166599 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-fmv8"
server_key: "94c0903d1300ce08e32b13d65a73aa25"
server_key_iv: "c126b0d852b34fd497c97972dece970e"
server_key_version: "encryptionkey@0"
I20260929 18:33:59.167169 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-fmv8"
server_key: "94c0903d1300ce08e32b13d65a73aa25"
server_key_iv: "c126b0d852b34fd497c97972dece970e"
server_key_version: "encryptionkey@0"
I20260929 18:33:59.170773 16117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 18:33:59.173257 16132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:59.174507 16117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:33:59.174654 16117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-fmv8"
server_key: "94c0903d1300ce08e32b13d65a73aa25"
server_key_iv: "c126b0d852b34fd497c97972dece970e"
server_key_version: "encryptionkey@0"
I20260929 18:33:59.174768 16117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:59.195397 16117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:59.198236 16117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:59.198436 16117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:59.199010 16117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:59.199801 16117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:59.199872 16117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:59.199940 16117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:59.199991 16117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:59.208531 16117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42967
I20260929 18:33:59.208577 16245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42967 every 8 connection(s)
I20260929 18:33:59.209388 16117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:59.213523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16117
I20260929 18:33:59.213641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:59.213937 392 external_mini_cluster.cc:1428] Setting key beeaba17392ae422c90139fc7059800f
Sep 29 18:33:59 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:33:59.222719 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.211135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32775 (local address 127.0.98.62:40839)
0929 18:33:59.211295 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.211300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.212523 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 18:33:59.213349 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.213352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.213397 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.213471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:59.214962 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.215728 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.217008 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.217218 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.217298 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.219500 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:59.219503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:59.219505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:59.219526 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:59.221086 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.221514 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.221517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.221518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.221588 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.221856 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.221858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.221859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.222047 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:59.222120 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.222518 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.222608 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:33:59.223555 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.211489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:32775)
0929 18:33:59.212380 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.212420 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.213144 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.213659 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.213669 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.214182 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:59.214821 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.214841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.215856 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.215859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.216870 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.216878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.217338 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.217340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.217469 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.217967 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:59.217981 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:59.219355 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:59.221187 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.221192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.221202 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.221431 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.221663 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.221665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.221666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.221781 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.222131 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:59.222135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:59.222330 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.222474 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.222686 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":545,"threads_started":1}
I20260929 18:33:59.224728 16246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:33:59.224946 16246 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:59.225406 16246 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
I20260929 18:33:59.226428 15787 ts_manager.cc:194] Registered new tserver with Master: 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967)
I20260929 18:33:59.227138 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32775
I20260929 18:33:59.227720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:59.236740 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.229832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.0.98.62:40839)
0929 18:33:59.229971 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.229975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.230129 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:33:59.230261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.230264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.230308 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.230388 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:59.231040 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.231527 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.232401 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.232619 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.232726 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.233934 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:59.233937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:59.233940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:59.233962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:59.235093 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.235494 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.235500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.235504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.235584 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.235865 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.235869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.235870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.236081 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:59.236249 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.236474 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.236583 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:33:59.239450 15786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:33:59.242477 15786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:33:59.257225 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.251143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.0.1:57540)
0929 18:33:59.251865 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.251879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.251982 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.252947 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.252952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.252971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:59.253085 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.253091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254237 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.254240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.255657 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.255668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.256654 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.256658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.256772 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.256811 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.256964 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.257083 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":55,"thread_start_us":118,"threads_started":1}
I20260929 18:33:59.257236 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.251712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.98.3:42967)
0929 18:33:59.252370 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.252386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.252408 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:33:59.252489 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.252497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.252662 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.252798 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:59.253193 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254124 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.255793 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.256528 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.256655 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.256791 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.256989 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.257083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":146,"threads_started":1}
I20260929 18:33:59.257587 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.251615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:55832)
0929 18:33:59.252656 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.252668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.252829 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.253493 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.253496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.253513 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:59.253617 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.253623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254861 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.254868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.256118 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.256128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.257121 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.257124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.257223 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.257245 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.257331 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.257430 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"spinlock_wait_cycles":1664,"thread_start_us":116,"threads_started":1}
I20260929 18:33:59.257807 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.250929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38457 (local address 127.0.0.1:35880)
0929 18:33:59.251712 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.251763 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.251902 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.252738 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.252747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.252774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:59.252921 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.252930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254161 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.254166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.255692 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.255702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.257408 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.257411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.257507 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.257524 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.257614 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.257683 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":124,"spinlock_wait_cycles":896,"thread_start_us":64,"threads_started":1}
I20260929 18:33:59.257874 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.251114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.0.98.2:38457)
0929 18:33:59.251518 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.251531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.252055 (+ 524us) server_negotiation.cc:407] Connection header received
0929 18:33:59.252134 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.252138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.252391 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.252547 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:59.253054 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254030 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.255819 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.256516 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.256627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.256769 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.257649 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.257739 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":170,"threads_started":1}
I20260929 18:33:59.260308 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.252068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.98.1:37269)
0929 18:33:59.252515 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.252531 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.252747 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:33:59.252982 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.253011 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.253188 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.253302 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:59.253740 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.254739 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.256245 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.256996 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.257120 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.257247 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.257783 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.257881 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":152,"threads_started":1}
I20260929 18:33:59.260761 16180 tablet_service.cc:1467] Processing CreateTablet for tablet 51208119b173445ab2ab199323347b87 (DEFAULT_TABLE table=test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:59.260761 16046 tablet_service.cc:1467] Processing CreateTablet for tablet 51208119b173445ab2ab199323347b87 (DEFAULT_TABLE table=test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:59.261888 16046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51208119b173445ab2ab199323347b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:59.261891 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51208119b173445ab2ab199323347b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:59.262966 15912 tablet_service.cc:1467] Processing CreateTablet for tablet 51208119b173445ab2ab199323347b87 (DEFAULT_TABLE table=test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:59.263725 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51208119b173445ab2ab199323347b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:59.269816 16263 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Bootstrap starting.
I20260929 18:33:59.270252 16264 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Bootstrap starting.
I20260929 18:33:59.270354 16265 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Bootstrap starting.
I20260929 18:33:59.272043 16264 tablet_bootstrap.cc:654] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:59.272204 16265 tablet_bootstrap.cc:654] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:59.272576 16263 tablet_bootstrap.cc:654] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:59.273195 16264 log.cc:824] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:59.273420 16265 log.cc:824] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:59.273427 16263 log.cc:824] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:59.275279 16263 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: No bootstrap required, opened a new log
I20260929 18:33:59.275279 16264 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: No bootstrap required, opened a new log
I20260929 18:33:59.275470 16264 ts_tablet_manager.cc:1397] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:33:59.275470 16263 ts_tablet_manager.cc:1397] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:59.275875 16265 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: No bootstrap required, opened a new log
I20260929 18:33:59.276081 16265 ts_tablet_manager.cc:1397] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:59.279071 16265 raft_consensus.cc:348] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.279397 16265 raft_consensus.cc:374] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:59.279460 16265 raft_consensus.cc:729] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569d50f3c7ce4b6181c370d1ec96ecf5, State: Initialized, Role: FOLLOWER
I20260929 18:33:59.279435 16264 raft_consensus.cc:348] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.279436 16263 raft_consensus.cc:348] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.279691 16263 raft_consensus.cc:374] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:59.279798 16264 raft_consensus.cc:374] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:59.279799 16263 raft_consensus.cc:729] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986d1c9a9da34f109edd050299afdae5, State: Initialized, Role: FOLLOWER
I20260929 18:33:59.279857 16264 raft_consensus.cc:729] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dbc565a91342b1b1356f075a22d654, State: Initialized, Role: FOLLOWER
I20260929 18:33:59.280086 16265 consensus_queue.cc:260] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.280267 16263 consensus_queue.cc:260] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.280269 16264 consensus_queue.cc:260] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.280956 15978 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
I20260929 18:33:59.281208 16265 ts_tablet_manager.cc:1428] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:33:59.281213 16264 ts_tablet_manager.cc:1428] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:33:59.281213 16263 ts_tablet_manager.cc:1428] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:33:59.281497 16246 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
I20260929 18:33:59.281536 16112 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
I20260929 18:33:59.361249 16269 raft_consensus.cc:482] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:59.361471 16269 raft_consensus.cc:504] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.362679 16269 leader_election.cc:288] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457), 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:33:59.367517 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.363042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48721 (local address 127.0.98.2:38457)
0929 18:33:59.363222 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.363230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.363575 (+ 345us) server_negotiation.cc:407] Connection header received
0929 18:33:59.363688 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.363690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.363730 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.363816 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:59.364064 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.364784 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.366273 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.366882 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.367034 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.367086 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.367316 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.367388 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 18:33:59.367517 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.363110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46467 (local address 127.0.98.1:37269)
0929 18:33:59.363230 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.363234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.363250 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:33:59.363396 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.363398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.363454 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.363523 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:59.363824 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.364649 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.366273 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.366882 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.367005 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.367074 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.367316 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.367388 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:33:59.367543 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.362927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.98.3:46467)
0929 18:33:59.363108 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.363133 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.363232 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.363571 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.363574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.363586 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:59.363723 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.363728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.364758 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.364760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.366145 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.366156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.367015 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.367018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.367174 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.367217 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.367287 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.367399 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":85}
I20260929 18:33:59.367551 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.362927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38457 (local address 127.0.98.3:48721)
0929 18:33:59.363453 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.363463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.363582 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.363825 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.363829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.363848 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:59.363960 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.363965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.364885 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.364888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.366145 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.366156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.367031 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.367034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.367159 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.367207 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.367287 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.367395 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":254,"thread_start_us":141,"threads_started":1}
I20260929 18:33:59.368227 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51208119b173445ab2ab199323347b87" candidate_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dbc565a91342b1b1356f075a22d654" is_pre_election: true
I20260929 18:33:59.368289 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51208119b173445ab2ab199323347b87" candidate_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986d1c9a9da34f109edd050299afdae5" is_pre_election: true
I20260929 18:33:59.368537 15932 raft_consensus.cc:2454] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569d50f3c7ce4b6181c370d1ec96ecf5 in term 0.
I20260929 18:33:59.368573 16066 raft_consensus.cc:2454] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569d50f3c7ce4b6181c370d1ec96ecf5 in term 0.
I20260929 18:33:59.369067 16136 leader_election.cc:302] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 99dbc565a91342b1b1356f075a22d654; no voters:
I20260929 18:33:59.369342 16269 raft_consensus.cc:2781] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:59.369535 16269 raft_consensus.cc:482] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:59.369606 16269 raft_consensus.cc:3037] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:59.370631 16269 raft_consensus.cc:504] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.370970 16269 leader_election.cc:288] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [CANDIDATE]: Term 1 election: Requested vote from peers 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457), 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:33:59.371332 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51208119b173445ab2ab199323347b87" candidate_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dbc565a91342b1b1356f075a22d654"
I20260929 18:33:59.371459 15932 raft_consensus.cc:3037] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:59.372298 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51208119b173445ab2ab199323347b87" candidate_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986d1c9a9da34f109edd050299afdae5"
I20260929 18:33:59.372437 16066 raft_consensus.cc:3037] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:59.372654 15932 raft_consensus.cc:2454] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569d50f3c7ce4b6181c370d1ec96ecf5 in term 1.
I20260929 18:33:59.373005 16136 leader_election.cc:302] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 99dbc565a91342b1b1356f075a22d654; no voters:
I20260929 18:33:59.373214 16269 raft_consensus.cc:2781] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:59.373440 16269 raft_consensus.cc:686] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 LEADER]: Becoming Leader. State: Replica: 569d50f3c7ce4b6181c370d1ec96ecf5, State: Running, Role: LEADER
I20260929 18:33:59.373786 16066 raft_consensus.cc:2454] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569d50f3c7ce4b6181c370d1ec96ecf5 in term 1.
I20260929 18:33:59.373806 16269 consensus_queue.cc:237] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:33:59.376880 15785 catalog_manager.cc:5495] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } health_report { overall_health: UNKNOWN } } }
I20260929 18:33:59.411425 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.408192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.3:42967)
0929 18:33:59.408329 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.408333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.408345 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:33:59.408417 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.408420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.408482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.408572 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:59.408789 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.409296 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.410085 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.410285 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.410359 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.410627 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:59.411033 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:59.411122 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.411179 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.411225 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:33:59.417115 15932 raft_consensus.cc:1261] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Refusing update from remote peer 569d50f3c7ce4b6181c370d1ec96ecf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:59.417115 16066 raft_consensus.cc:1261] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Refusing update from remote peer 569d50f3c7ce4b6181c370d1ec96ecf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:59.417788 16269 consensus_queue.cc:1035] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:59.418022 16273 consensus_queue.cc:1035] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:59.424505 16276 mvcc.cc:205] Tried to move back new op lower bound from 7334735214245695488 to 7334735214077603840. Current Snapshot: MvccSnapshot[applied={T|T < 7334735214245695488}]
I20260929 18:33:59.425361 16278 mvcc.cc:205] Tried to move back new op lower bound from 7334735214245695488 to 7334735214077603840. Current Snapshot: MvccSnapshot[applied={T|T < 7334735214245695488}]
I20260929 18:33:59.426923 16277 mvcc.cc:205] Tried to move back new op lower bound from 7334735214245695488 to 7334735214077603840. Current Snapshot: MvccSnapshot[applied={T|T < 7334735214245695488}]
I20260929 18:33:59.433737 15787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:59.433874 15787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:59.436533 15787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Sending DeleteTablet for 3 replicas of tablet 51208119b173445ab2ab199323347b87
I20260929 18:33:59.437239 16046 tablet_service.cc:1514] Processing DeleteTablet for tablet 51208119b173445ab2ab199323347b87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35880
I20260929 18:33:59.437255 16180 tablet_service.cc:1514] Processing DeleteTablet for tablet 51208119b173445ab2ab199323347b87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57540
I20260929 18:33:59.437409 15912 tablet_service.cc:1514] Processing DeleteTablet for tablet 51208119b173445ab2ab199323347b87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55832
I20260929 18:33:59.437922 16286 tablet_replica.cc:330] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: stopping tablet replica
I20260929 18:33:59.438292 16286 raft_consensus.cc:2229] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:59.438449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15849
I20260929 18:33:59.439483 16288 tablet_replica.cc:330] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: stopping tablet replica
I20260929 18:33:59.439797 16288 raft_consensus.cc:2229] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:33:59.440265 16288 raft_consensus.cc:2258] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:59.441793 16288 ts_tablet_manager.cc:1905] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:33:59.443789 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37269 (error 108)
I20260929 18:33:59.443835 16287 tablet_replica.cc:330] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: stopping tablet replica
I20260929 18:33:59.444121 16287 raft_consensus.cc:2229] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:59.444450 16287 raft_consensus.cc:2258] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:59.444629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15983
I20260929 18:33:59.445325 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.444921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:55846)
0929 18:33:59.445152 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.445226 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260929 18:33:59.445443 15774 catalog_manager.cc:4584] TS 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51208119b173445ab2ab199323347b87: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
I20260929 18:33:59.445950 16288 ts_tablet_manager.cc:1918] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:59.446072 16288 log.cc:1196] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/51208119b173445ab2ab199323347b87
I20260929 18:33:59.446457 16288 ts_tablet_manager.cc:1939] T 51208119b173445ab2ab199323347b87 P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting consensus metadata
I20260929 18:33:59.447640 15772 catalog_manager.cc:4841] TS 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): tablet 51208119b173445ab2ab199323347b87 (table test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]) successfully deleted
I20260929 18:33:59.450748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16117
I20260929 18:33:59.451030 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.450750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38457 (local address 127.0.0.1:35892)
0929 18:33:59.450866 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.450929 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 18:33:59.456287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 15755
I20260929 18:34:19.462563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--webserver_interface=127.0.98.62
--webserver_port=37135
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:19.571326 16290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:19.571573 16290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:19.571669 16290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:19.575752 16290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:34:19.575809 16290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:34:19.575825 16290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:34:19.575839 16290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:19.575886 16290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:19.575925 16290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:19.575950 16290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:19.575982 16290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:19.581162 16290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:19.582170 16290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:19.583145 16290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:34:19.596076 16290 server_base.cc:1034] running on GCE node
W20260929 18:34:19.596022 16297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:19.596019 16299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:19.596313 16296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:19.596908 16290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:19.598012 16290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:19.599203 16290 hybrid_clock.cc:648] HybridClock initialized: now 1790706859599188 us; error 30 us; skew 500 ppm
Sep 29 18:34:19 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:19.606722 16290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:34:19.607686 16290 webserver.cc:460] Webserver started at http://127.0.98.62:37135/ using document root <none> and password file <none>
I20260929 18:34:19.608420 16290 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:19.608542 16290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:19.612740 16290 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:34:19.614807 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:19.616010 16290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:19.616187 16290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0d8c403b5ec4420196cb1334e445a2c9"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "22122886a331716acd058db595c454ef"
server_key_iv: "388acbc60546eeab59c9d8dfbbbfed76"
server_key_version: "encryptionkey@0"
I20260929 18:34:19.616796 16290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:19.645205 16290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:19.648090 16290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:19.648317 16290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:19.655232 16290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40839
I20260929 18:34:19.655521 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40839 every 8 connection(s)
I20260929 18:34:19.656366 16290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:19.659159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16290
I20260929 18:34:19.659636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37269
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=35879
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:19.661947 16359 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:34:19.663636 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap starting.
I20260929 18:34:19.680411 16359 log.cc:824] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:19.688776 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:19.689163 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap complete.
I20260929 18:34:19.693167 16359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:19.693535 16359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Initialized, Role: FOLLOWER
I20260929 18:34:19.694032 16359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:19.694154 16359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:19.694242 16359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:19.694353 16359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:34:19.696213 16359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:19.696540 16359 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8c403b5ec4420196cb1334e445a2c9; no voters:
I20260929 18:34:19.696849 16359 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:34:19.696926 16363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:34:19.697180 16363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 LEADER]: Becoming Leader. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Running, Role: LEADER
I20260929 18:34:19.697512 16363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:19.698019 16359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:19.698774 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:34:19.698909 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:19.699576 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8c403b5ec4420196cb1334e445a2c9. Latest consensus state: current_term: 2 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:34:19.699692 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:19.700158 16372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:34:19.703779 16372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:34:19.704416 16372 catalog_manager.cc:1259] Loaded cluster ID: c2bb05a427784640995c2950bd8984be
I20260929 18:34:19.704491 16372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:34:19.706156 16372 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:34:19.706732 16372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Loaded TSK: 0
I20260929 18:34:19.718111 16372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Generated new TSK 1
I20260929 18:34:19.718353 16372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:34:19.781361 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:19.781694 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:19.781785 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:19.785853 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:19.785928 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:19.786012 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:34:19.791570 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35879
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:19.792729 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:19.793723 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:19.803397 16388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:19.803382 16390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:19.803664 16361 server_base.cc:1034] running on GCE node
W20260929 18:34:19.803519 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:19.804188 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:19.804932 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:19.806123 16361 hybrid_clock.cc:648] HybridClock initialized: now 1790706859806098 us; error 34 us; skew 500 ppm
Sep 29 18:34:19 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:19.809296 16361 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:34:19.810155 16361 webserver.cc:460] Webserver started at http://127.0.98.1:35879/ using document root <none> and password file <none>
I20260929 18:34:19.810707 16361 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:19.810825 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:19.814651 16361 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:34:19.816931 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:19.818192 16361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:19.818332 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "99dbc565a91342b1b1356f075a22d654"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "c8921f5866fefdd3614b492a64efb61c"
server_key_iv: "5ec170568fac390bc65601f1aa182ba7"
server_key_version: "encryptionkey@0"
I20260929 18:34:19.818862 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:19.845206 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:19.848444 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:19.848654 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:19.849267 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:19.850446 16404 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:34:19.854516 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:34:19.854579 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 18:34:19.854650 16361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:34:19.856350 16361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:34:19.856408 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:19.856658 16404 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Bootstrap starting.
I20260929 18:34:19.867409 16361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260929 18:34:19.867451 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260929 18:34:19.868688 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:34:19.876771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16361
I20260929 18:34:19.877208 16404 log.cc:824] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:19.877285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38457
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=43767
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:34:19 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:19.885229 16404 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:19.885733 16404 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Bootstrap complete.
I20260929 18:34:19.886407 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:19.870877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:48193)
0929 18:34:19.871403 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:19.871443 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:19.872330 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:19.873423 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:19.873431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:19.873850 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:19.874284 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:19.874297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.875581 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.875584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:19.876529 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:19.876538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.877351 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.877355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:19.877541 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:19.878255 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:19.878275 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:19.880141 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:19.882842 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:19.882847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:19.882860 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:19.883179 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:19.883625 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:19.883628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:19.883630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:19.883785 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:19.884603 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:19.884609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:19.884884 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 18:34:19.885068 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:34:19.885391 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":171,"threads_started":1}
I20260929 18:34:19.886569 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:19.870840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48193 (local address 127.0.98.62:40839)
0929 18:34:19.871241 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:19.871266 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:19.871562 (+ 296us) server_negotiation.cc:407] Connection header received
0929 18:34:19.872757 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:19.872767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:19.873053 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:19.873373 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:19.874454 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.875413 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.876659 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.877192 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.877387 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:19.880365 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:19.880375 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:19.880395 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:19.880435 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:19.882708 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:19.883323 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:19.883332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:19.883340 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:19.883508 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:19.883904 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:19.883908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:19.883911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:19.884460 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:19.884619 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:19.885254 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 18:34:19.885603 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
I20260929 18:34:19.887143 16404 ts_tablet_manager.cc:1397] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260929 18:34:19.889547 16404 raft_consensus.cc:348] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:34:19.889972 16404 raft_consensus.cc:729] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dbc565a91342b1b1356f075a22d654, State: Initialized, Role: FOLLOWER
I20260929 18:34:19.890488 16404 consensus_queue.cc:260] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:34:19.890599 16512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:34:19.890830 16512 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:19.891337 16512 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
I20260929 18:34:19.891387 16404 ts_tablet_manager.cc:1428] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 18:34:19.893940 16323 ts_manager.cc:194] Registered new tserver with Master: 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:34:19.897931 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48193
I20260929 18:34:19.900694 16512 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
Sep 29 18:34:19 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 18:34:19.907963 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:19.897247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:60476)
0929 18:34:19.897956 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:19.898001 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:19.898156 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:19.898684 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:19.898689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:19.899132 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:19.899258 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:19.899267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.900872 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.900875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:19.901638 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:19.901648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.902206 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.902222 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:19.902384 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:19.902964 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:19.902976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:19.904470 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:19.906248 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:19.906258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:19.906263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:19.906489 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:19.906786 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:19.906791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:19.906793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:19.906875 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:19.907311 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:19.907317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:19.907437 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:34:19.907592 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 18:34:19.907783 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":121,"threads_started":1}
I20260929 18:34:19.907980 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:19.897288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.0.98.1:37269)
0929 18:34:19.897808 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:19.897819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:19.898262 (+ 443us) server_negotiation.cc:407] Connection header received
0929 18:34:19.898317 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:19.898320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:19.898517 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:19.898685 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:19.899386 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.900690 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.901757 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:19.902101 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:19.902216 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:19.904617 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:19.904625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:19.904634 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:19.904653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:19.906136 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:19.906614 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:19.906618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:19.906620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:19.906692 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:19.906973 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:19.906976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:19.906978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:19.907198 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:19.907337 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:19.907671 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 18:34:19.907823 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":425,"thread_start_us":101,"threads_started":1}
I20260929 18:34:19.909224 16446 tablet_service.cc:1514] Processing DeleteTablet for tablet 51208119b173445ab2ab199323347b87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60476
I20260929 18:34:19.910210 16521 tablet_replica.cc:330] stopping tablet replica
I20260929 18:34:19.910501 16521 raft_consensus.cc:2229] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:19.910743 16521 raft_consensus.cc:2258] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:19.911661 16521 ts_tablet_manager.cc:1905] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:19.915055 16521 ts_tablet_manager.cc:1918] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:19.915166 16521 log.cc:1196] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal/wals/51208119b173445ab2ab199323347b87
I20260929 18:34:19.915485 16521 ts_tablet_manager.cc:1939] T 51208119b173445ab2ab199323347b87 P 99dbc565a91342b1b1356f075a22d654: Deleting consensus metadata
I20260929 18:34:19.917081 16310 catalog_manager.cc:4841] TS 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269): tablet 51208119b173445ab2ab199323347b87 (table test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]) successfully deleted
W20260929 18:34:19.997860 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:19.998140 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:19.998229 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:20.002431 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:20.002485 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:20.002578 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:34:20.008044 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43767
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:20.009440 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:20.010475 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:20.020113 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:20.020200 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:20.020210 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:20.020259 16517 server_base.cc:1034] running on GCE node
I20260929 18:34:20.020736 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:20.021442 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:20.022643 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790706860022610 us; error 49 us; skew 500 ppm
Sep 29 18:34:20 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:20.025449 16517 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:34:20.026371 16517 webserver.cc:460] Webserver started at http://127.0.98.2:43767/ using document root <none> and password file <none>
I20260929 18:34:20.026927 16517 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:20.027038 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:20.030884 16517 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:20.033059 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:20.034289 16517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:34:20.034421 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "986d1c9a9da34f109edd050299afdae5"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "640a85a3e6e74067d8c08497bb2e41b9"
server_key_iv: "0f13f242bcede8654c19e3ec526f3a8b"
server_key_version: "encryptionkey@0"
I20260929 18:34:20.034989 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:20.061719 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:20.065040 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:20.065265 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:20.065881 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:20.066994 16543 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:34:20.071116 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:34:20.071215 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:34:20.071305 16517 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:34:20.073227 16517 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:34:20.073297 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260929 18:34:20.073601 16543 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Bootstrap starting.
I20260929 18:34:20.083050 16517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38457
I20260929 18:34:20.083335 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38457 every 8 connection(s)
I20260929 18:34:20.084084 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:20.085462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16517
I20260929 18:34:20.085908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42967
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=35861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:20.093910 16543 log.cc:824] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:34:20 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:20.101516 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.086115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34193 (local address 127.0.98.62:40839)
0929 18:34:20.086258 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.086263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.086956 (+ 693us) server_negotiation.cc:407] Connection header received
0929 18:34:20.088363 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.088367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.088424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.088532 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:20.090193 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.091114 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.093113 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.093463 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.093585 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.096808 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:20.096812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:20.096815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:20.096847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:20.098823 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.099762 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.099766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.099768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.099867 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.100269 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.100275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.100279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.100515 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:20.100615 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.101213 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.101346 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:34:20.102396 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.086278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:34193)
0929 18:34:20.086791 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.086846 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.088070 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.088735 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.088748 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.089273 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:20.089987 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.090011 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.091288 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.091294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.092965 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.092981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.093624 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.093631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.093900 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.094612 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:20.094628 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:20.096603 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:20.099045 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:20.099051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:20.099065 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:20.099639 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:20.099965 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:20.099968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:20.099970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:20.100169 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:20.100626 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:20.100633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:20.100937 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.101128 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.101457 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
I20260929 18:34:20.102566 16543 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:20.103015 16543 tablet_bootstrap.cc:492] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Bootstrap complete.
I20260929 18:34:20.103605 16651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:34:20.103837 16651 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:20.104351 16543 ts_tablet_manager.cc:1397] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20260929 18:34:20.104377 16651 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
I20260929 18:34:20.106606 16323 ts_manager.cc:194] Registered new tserver with Master: 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
I20260929 18:34:20.106814 16543 raft_consensus.cc:348] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:34:20.107344 16543 raft_consensus.cc:729] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986d1c9a9da34f109edd050299afdae5, State: Initialized, Role: FOLLOWER
I20260929 18:34:20.107815 16543 consensus_queue.cc:260] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } } peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260929 18:34:20.107887 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34193
I20260929 18:34:20.108851 16543 ts_tablet_manager.cc:1428] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260929 18:34:20.110080 16651 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
I20260929 18:34:20.114612 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.108813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.0.98.2:38457)
0929 18:34:20.110486 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.110499 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.110524 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:34:20.110594 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.110599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.110806 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.110981 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:20.111221 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.112103 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.113579 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.114068 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.114149 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.114278 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.114453 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.114523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":110,"threads_started":1}
I20260929 18:34:20.114627 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.108333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38457 (local address 127.0.0.1:50946)
0929 18:34:20.108590 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.108605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.108678 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.110978 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.110981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.111002 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:20.111102 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.111108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.112234 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.112238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.113443 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.113453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.114181 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.114184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.114282 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.114335 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.114413 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.114511 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 18:34:20.115746 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet 51208119b173445ab2ab199323347b87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50946
I20260929 18:34:20.116532 16658 tablet_replica.cc:330] stopping tablet replica
I20260929 18:34:20.116813 16658 raft_consensus.cc:2229] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:20.117033 16658 raft_consensus.cc:2258] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:20.117975 16658 ts_tablet_manager.cc:1905] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:20.122229 16658 ts_tablet_manager.cc:1918] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:20.122320 16658 log.cc:1196] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/51208119b173445ab2ab199323347b87
I20260929 18:34:20.122594 16658 ts_tablet_manager.cc:1939] T 51208119b173445ab2ab199323347b87 P 986d1c9a9da34f109edd050299afdae5: Deleting consensus metadata
I20260929 18:34:20.124221 16309 catalog_manager.cc:4841] TS 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): tablet 51208119b173445ab2ab199323347b87 (table test-table [id=e7bc6e30fa78419cbdd27e9b2b96864d]) successfully deleted
W20260929 18:34:20.204609 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:20.204861 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:20.204905 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:20.208992 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:20.209045 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:20.209132 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:34:20.214329 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35861
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:20.215360 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:20.216524 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:20.225898 16666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:20.225904 16663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:20.225911 16664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:20.226225 16654 server_base.cc:1034] running on GCE node
I20260929 18:34:20.226647 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:20.227229 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:20.228439 16654 hybrid_clock.cc:648] HybridClock initialized: now 1790706860228419 us; error 31 us; skew 500 ppm
Sep 29 18:34:20 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:20.231248 16654 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:34:20.232095 16654 webserver.cc:460] Webserver started at http://127.0.98.3:35861/ using document root <none> and password file <none>
I20260929 18:34:20.232681 16654 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:20.232787 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:20.236531 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:34:20.238641 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:20.239943 16654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:20.240087 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-fmv8"
server_key: "94c0903d1300ce08e32b13d65a73aa25"
server_key_iv: "c126b0d852b34fd497c97972dece970e"
server_key_version: "encryptionkey@0"
I20260929 18:34:20.240672 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:20.258241 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:20.261125 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:20.261322 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:20.261871 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:20.262640 16654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:20.262708 16654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:20.262776 16654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:20.262827 16654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:20.271726 16654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42967
I20260929 18:34:20.271750 16786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42967 every 8 connection(s)
I20260929 18:34:20.272706 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:20.273211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16654
I20260929 18:34:20.282738 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.273941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.98.62:40839)
0929 18:34:20.274111 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.274116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.274409 (+ 293us) server_negotiation.cc:407] Connection header received
0929 18:34:20.274614 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.274617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.274669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.274752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:20.275704 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.276329 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.277634 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.277883 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.277988 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.279616 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:20.279619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:20.279621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:20.279676 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:20.281101 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.281565 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.281568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.281569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.281644 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.281927 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.281930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.281931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.282149 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:20.282337 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.282467 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.282587 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 18:34:20 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:20.288781 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.274450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41535 (local address 127.0.98.62:40839)
0929 18:34:20.274736 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.274741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.275663 (+ 922us) server_negotiation.cc:407] Connection header received
0929 18:34:20.277917 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.277923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.278027 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.278122 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:20.279711 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.280586 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.282067 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.282377 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.282481 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.285300 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:20.285303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:20.285306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:20.285330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:20.286676 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.287192 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.287196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.287198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.287286 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.287635 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.287642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.287644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.287817 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:20.287940 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.288490 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.288610 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260929 18:34:20.289557 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.274985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:41535)
0929 18:34:20.275507 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.275548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.277723 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.278333 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.278345 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.278861 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:20.279557 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.279581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.280718 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.280721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.281947 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.281956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.282552 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.282560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.282755 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.283377 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:20.283392 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:20.285135 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:20.286798 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:20.286804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:20.286817 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:20.287084 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:20.287393 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:20.287398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:20.287404 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:20.287532 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:20.287941 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:20.287949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:20.288236 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.288438 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.288746 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260929 18:34:20.290659 16787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40839
I20260929 18:34:20.290906 16787 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:20.291405 16787 heartbeater.cc:511] Master 127.0.98.62:40839 requested a full tablet report, sending...
I20260929 18:34:20.292498 16323 ts_manager.cc:194] Registered new tserver with Master: 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967)
I20260929 18:34:20.293100 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41535
I20260929 18:34:20.296486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:20.300732 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.297141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.0.98.62:40839)
0929 18:34:20.297267 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.297271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.297455 (+ 184us) server_negotiation.cc:407] Connection header received
0929 18:34:20.297637 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.297640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.297728 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.297814 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:34:20.298075 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.298670 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.299631 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.299859 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.299941 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.300284 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:34:20.300403 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:34:20.300614 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38644: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:34:20.300802 16790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38644: token expired
W20260929 18:34:20.301215 16249 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40839)
I20260929 18:34:20.308318 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.301576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.98.62:40839)
0929 18:34:20.301684 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.301687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.301698 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:34:20.301768 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.301770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.301834 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.301895 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:20.302472 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.302928 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.303761 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.303964 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.304045 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.305299 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:20.305302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:20.305304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:20.305324 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:20.306837 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.307197 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.307200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.307201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.307269 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:20.307528 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:20.307532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:20.307534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:20.307718 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:20.307826 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.308010 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.308104 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:34:20.310599 16323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:34:20.313803 16323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:34:20.321025 16446 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9a59c4e8b40028dbe240183499dba (DEFAULT_TABLE table=test-table [id=cdd34bf7d20646bb9f09efc29d2b0f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:20.321604 16585 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9a59c4e8b40028dbe240183499dba (DEFAULT_TABLE table=test-table [id=cdd34bf7d20646bb9f09efc29d2b0f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:20.321748 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9a59c4e8b40028dbe240183499dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:20.322154 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9a59c4e8b40028dbe240183499dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:20.324709 16404 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Bootstrap starting.
I20260929 18:34:20.324810 16543 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Bootstrap starting.
I20260929 18:34:20.325554 16543 tablet_bootstrap.cc:654] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:20.325554 16404 tablet_bootstrap.cc:654] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:20.326027 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.320130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.0.1:41072)
0929 18:34:20.320258 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.320273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.320353 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.321464 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.321482 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.321508 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:20.321598 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.321603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.323104 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.323107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.324312 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.324323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.325723 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.325726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.325813 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.325835 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.325875 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.325932 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:34:20.326879 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.320650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.98.3:42967)
0929 18:34:20.320957 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.320976 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.320996 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:34:20.321061 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.321065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.321297 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.321481 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:20.321741 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.322987 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.324491 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.325559 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.326321 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.326580 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.326637 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.326711 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":133,"threads_started":1}
I20260929 18:34:20.326924 16543 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: No bootstrap required, opened a new log
I20260929 18:34:20.327024 16543 ts_tablet_manager.cc:1397] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:34:20.327180 16404 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: No bootstrap required, opened a new log
I20260929 18:34:20.327241 16404 ts_tablet_manager.cc:1397] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:34:20.327561 16404 raft_consensus.cc:348] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.327646 16404 raft_consensus.cc:374] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:20.327672 16404 raft_consensus.cc:729] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dbc565a91342b1b1356f075a22d654, State: Initialized, Role: FOLLOWER
I20260929 18:34:20.327693 16543 raft_consensus.cc:348] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.327807 16543 raft_consensus.cc:374] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:20.327847 16543 raft_consensus.cc:729] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986d1c9a9da34f109edd050299afdae5, State: Initialized, Role: FOLLOWER
I20260929 18:34:20.327803 16404 consensus_queue.cc:260] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.327966 16543 consensus_queue.cc:260] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.328126 16404 ts_tablet_manager.cc:1428] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:34:20.328730 16543 ts_tablet_manager.cc:1428] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:20.330096 16721 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9a59c4e8b40028dbe240183499dba (DEFAULT_TABLE table=test-table [id=cdd34bf7d20646bb9f09efc29d2b0f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:20.331207 16721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9a59c4e8b40028dbe240183499dba. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 18:34:20.335381 16652 tablet.cc:2369] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:20.338227 16799 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Bootstrap starting.
I20260929 18:34:20.340517 16799 tablet_bootstrap.cc:654] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:20.341926 16799 log.cc:824] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:20.343981 16799 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: No bootstrap required, opened a new log
I20260929 18:34:20.344247 16799 ts_tablet_manager.cc:1397] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:34:20.347311 16799 raft_consensus.cc:348] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.347540 16799 raft_consensus.cc:374] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:20.347575 16799 raft_consensus.cc:729] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569d50f3c7ce4b6181c370d1ec96ecf5, State: Initialized, Role: FOLLOWER
I20260929 18:34:20.347971 16799 consensus_queue.cc:260] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.348613 16787 heartbeater.cc:503] Master 127.0.98.62:40839 was elected leader, sending a full tablet report...
I20260929 18:34:20.348865 16799 ts_tablet_manager.cc:1428] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 18:34:20.369841 16513 tablet.cc:2369] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:20.378625 16656 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:20.378800 16656 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.379894 16656 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:34:20.384717 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.380099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.98.2:40143)
0929 18:34:20.380337 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.380354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.380459 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.380769 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.380771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.380784 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:20.380931 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.380938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.381962 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.381965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.383374 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.383382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.384278 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.384282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.384403 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.384428 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.384497 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.384574 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":109}
I20260929 18:34:20.384845 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.380299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44103 (local address 127.0.98.1:37269)
0929 18:34:20.380432 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.380436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.380612 (+ 176us) server_negotiation.cc:407] Connection header received
0929 18:34:20.380701 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.380707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.380772 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.380872 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:20.381181 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.381974 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.383521 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.384240 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.384325 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.384491 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.384648 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.384722 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:34:20.384970 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.380207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40143 (local address 127.0.98.3:42967)
0929 18:34:20.380319 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.380323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.380436 (+ 113us) server_negotiation.cc:407] Connection header received
0929 18:34:20.380603 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.380607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.380666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.380745 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:20.381070 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.381851 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.383521 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.384132 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.384265 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.384473 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.384515 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.384586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:34:20.385213 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.380178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.98.2:44103)
0929 18:34:20.380534 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.380548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:20.380630 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:20.380895 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:20.380898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:20.380910 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:20.381076 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.381082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.382081 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.382084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.383366 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:20.383382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.384383 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.384386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:20.384502 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.384519 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:34:20.384968 (+ 449us) client_negotiation.cc:240] Negotiation successful
0929 18:34:20.385049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":31,"thread_start_us":99,"threads_started":1}
I20260929 18:34:20.385579 16741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9a59c4e8b40028dbe240183499dba" candidate_uuid: "986d1c9a9da34f109edd050299afdae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" is_pre_election: true
I20260929 18:34:20.385820 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9a59c4e8b40028dbe240183499dba" candidate_uuid: "986d1c9a9da34f109edd050299afdae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dbc565a91342b1b1356f075a22d654" is_pre_election: true
I20260929 18:34:20.385908 16741 raft_consensus.cc:2454] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986d1c9a9da34f109edd050299afdae5 in term 0.
I20260929 18:34:20.386080 16466 raft_consensus.cc:2454] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986d1c9a9da34f109edd050299afdae5 in term 0.
I20260929 18:34:20.386426 16538 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5; no voters:
I20260929 18:34:20.386695 16656 raft_consensus.cc:2781] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:34:20.386790 16656 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:34:20.386852 16656 raft_consensus.cc:3037] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:20.387917 16656 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.388285 16656 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [CANDIDATE]: Term 1 election: Requested vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269)
I20260929 18:34:20.388624 16741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9a59c4e8b40028dbe240183499dba" candidate_uuid: "986d1c9a9da34f109edd050299afdae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
I20260929 18:34:20.388624 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9a59c4e8b40028dbe240183499dba" candidate_uuid: "986d1c9a9da34f109edd050299afdae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99dbc565a91342b1b1356f075a22d654"
I20260929 18:34:20.388744 16466 raft_consensus.cc:3037] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:20.388746 16741 raft_consensus.cc:3037] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:20.389954 16741 raft_consensus.cc:2454] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986d1c9a9da34f109edd050299afdae5 in term 1.
I20260929 18:34:20.389954 16466 raft_consensus.cc:2454] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986d1c9a9da34f109edd050299afdae5 in term 1.
I20260929 18:34:20.390241 16538 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5; no voters:
I20260929 18:34:20.390396 16656 raft_consensus.cc:2781] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:20.390606 16656 raft_consensus.cc:686] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 1 LEADER]: Becoming Leader. State: Replica: 986d1c9a9da34f109edd050299afdae5, State: Running, Role: LEADER
I20260929 18:34:20.390857 16656 consensus_queue.cc:237] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:20.393960 16323 catalog_manager.cc:5495] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 986d1c9a9da34f109edd050299afdae5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "986d1c9a9da34f109edd050299afdae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } health_report { overall_health: HEALTHY } } }
I20260929 18:34:20.408339 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.405138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.0.98.2:38457)
0929 18:34:20.405254 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:20.405257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:20.405269 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:34:20.405371 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:20.405373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:20.405423 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:20.405510 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:34:20.405722 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.406223 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.407060 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:20.407272 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:20.407351 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:20.407572 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:34:20.407964 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:34:20.408072 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:20.408138 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:34:20.408234 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:34:20.412106 16466 raft_consensus.cc:1261] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Refusing update from remote peer 986d1c9a9da34f109edd050299afdae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:20.412122 16741 raft_consensus.cc:1261] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Refusing update from remote peer 986d1c9a9da34f109edd050299afdae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:20.412874 16656 consensus_queue.cc:1035] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:20.413098 16803 consensus_queue.cc:1035] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:20.418619 16807 mvcc.cc:205] Tried to move back new op lower bound from 7334735300241907712 to 7334735300163440640. Current Snapshot: MvccSnapshot[applied={T|T < 7334735300241907712}]
I20260929 18:34:20.420148 16806 mvcc.cc:205] Tried to move back new op lower bound from 7334735300241907712 to 7334735300163440640. Current Snapshot: MvccSnapshot[applied={T|T < 7334735300241907712}]
I20260929 18:34:20.428028 16321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:20.428368 16321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:20.430670 16321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Sending DeleteTablet for 3 replicas of tablet 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:20.431288 16446 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9a59c4e8b40028dbe240183499dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60476
I20260929 18:34:20.431455 16721 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9a59c4e8b40028dbe240183499dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260929 18:34:20.431665 16816 tablet_replica.cc:330] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: stopping tablet replica
I20260929 18:34:20.431813 16816 raft_consensus.cc:2229] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:20.431957 16816 raft_consensus.cc:2258] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:20.432017 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9a59c4e8b40028dbe240183499dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50946
I20260929 18:34:20.432176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16361
I20260929 18:34:20.432235 16658 tablet_replica.cc:330] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: stopping tablet replica
I20260929 18:34:20.432372 16658 raft_consensus.cc:2229] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:34:20.432664 16658 raft_consensus.cc:2258] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:20.434010 16658 ts_tablet_manager.cc:1905] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:20.436370 16817 tablet_replica.cc:330] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: stopping tablet replica
I20260929 18:34:20.436575 16658 ts_tablet_manager.cc:1918] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:20.436656 16658 log.cc:1196] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:20.436918 16658 ts_tablet_manager.cc:1939] T 09d9a59c4e8b40028dbe240183499dba P 986d1c9a9da34f109edd050299afdae5: Deleting consensus metadata
I20260929 18:34:20.436882 16817 raft_consensus.cc:2229] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:20.437317 16817 raft_consensus.cc:2258] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:20.437588 16309 catalog_manager.cc:4841] TS 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): tablet 09d9a59c4e8b40028dbe240183499dba (table test-table [id=cdd34bf7d20646bb9f09efc29d2b0f0c]) successfully deleted
I20260929 18:34:20.438948 16817 ts_tablet_manager.cc:1905] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:20.439553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16517
W20260929 18:34:20.440498 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37269 (error 108)
I20260929 18:34:20.441615 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:20.441261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:60490)
0929 18:34:20.441410 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:20.441532 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 18:34:20.441771 16310 catalog_manager.cc:4584] TS 99dbc565a91342b1b1356f075a22d654 (127.0.98.1:37269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09d9a59c4e8b40028dbe240183499dba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
I20260929 18:34:20.442947 16817 ts_tablet_manager.cc:1918] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:20.443061 16817 log.cc:1196] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:20.443377 16817 ts_tablet_manager.cc:1939] T 09d9a59c4e8b40028dbe240183499dba P 569d50f3c7ce4b6181c370d1ec96ecf5: Deleting consensus metadata
I20260929 18:34:20.444635 16308 catalog_manager.cc:4841] TS 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): tablet 09d9a59c4e8b40028dbe240183499dba (table test-table [id=cdd34bf7d20646bb9f09efc29d2b0f0c]) successfully deleted
I20260929 18:34:20.446908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16654
I20260929 18:34:20.454435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16290
I20260929 18:34:40.460939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--webserver_interface=127.0.98.62
--webserver_port=37135
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:40.569700 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:40.569948 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:40.569988 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:40.573997 16819 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:34:40.574050 16819 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:34:40.574066 16819 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:34:40.574080 16819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:40.574098 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:40.574116 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:40.574132 16819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:40.574194 16819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:40.579263 16819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:40.580302 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:40.581319 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:34:40.590404 16819 server_base.cc:1034] running on GCE node
W20260929 18:34:40.590323 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:40.590454 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:40.590320 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:40.591112 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:40.592308 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:40.593497 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790706880593481 us; error 36 us; skew 500 ppm
Sep 29 18:34:40 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:40.596359 16819 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:34:40.597296 16819 webserver.cc:460] Webserver started at http://127.0.98.62:37135/ using document root <none> and password file <none>
I20260929 18:34:40.597863 16819 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:40.597955 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:40.601943 16819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:34:40.604929 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:40.606547 16819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:40.606669 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0d8c403b5ec4420196cb1334e445a2c9"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "22122886a331716acd058db595c454ef"
server_key_iv: "388acbc60546eeab59c9d8dfbbbfed76"
server_key_version: "encryptionkey@0"
I20260929 18:34:40.607410 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:40.626391 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:40.629657 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:40.629810 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:40.636775 16819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40839
I20260929 18:34:40.637051 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40839 every 8 connection(s)
I20260929 18:34:40.637841 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:40.642035 16887 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:34:40.643590 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap starting.
I20260929 18:34:40.647688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16819
I20260929 18:34:40.648185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37269
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=35879
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:40.661564 16887 log.cc:824] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:40.672353 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:40.672719 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: Bootstrap complete.
I20260929 18:34:40.676672 16887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:40.677021 16887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Initialized, Role: FOLLOWER
I20260929 18:34:40.677529 16887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:40.677652 16887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:40.677743 16887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:40.677857 16887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:34:40.679651 16887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:40.679968 16887 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8c403b5ec4420196cb1334e445a2c9; no voters:
I20260929 18:34:40.680322 16887 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:34:40.680397 16891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:34:40.680603 16891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [term 3 LEADER]: Becoming Leader. State: Replica: 0d8c403b5ec4420196cb1334e445a2c9, State: Running, Role: LEADER
I20260929 18:34:40.680878 16891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } }
I20260929 18:34:40.681494 16887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:40.681733 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8c403b5ec4420196cb1334e445a2c9. Latest consensus state: current_term: 3 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:34:40.681859 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:40.683069 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0d8c403b5ec4420196cb1334e445a2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8c403b5ec4420196cb1334e445a2c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40839 } } }
I20260929 18:34:40.683246 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:40.686631 16907 catalog_manager.cc:1259] Loaded cluster ID: c2bb05a427784640995c2950bd8984be
I20260929 18:34:40.686695 16907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: loading cluster ID for follower catalog manager: success
I20260929 18:34:40.689090 16907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d8c403b5ec4420196cb1334e445a2c9: acquiring CA information for follower catalog manager: success
F20260929 18:34:40.690183 16907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f549b5919ad google::LogMessage::Fail() at ??:0
@ 0x7f549b59488a google::LogMessage::SendToLog() at ??:0
@ 0x7f549b5914de google::LogMessage::Flush() at ??:0
@ 0x7f549b592a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f549e25518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f54a5ef6989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f54a5ed3023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f54a5ed354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f54a5ecc652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f54a5ecb9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f54a5f12541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f549d7d4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f549ee186db start_thread at ??:0
@ 0x7f549a08f61f clone at ??:0
W20260929 18:34:40.782347 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:40.782598 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:40.782639 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:40.786783 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:40.786842 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:40.786900 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:34:40.792073 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35879
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:40.793136 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:40.794163 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:40.804935 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:40.804941 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:40.805225 16889 server_base.cc:1034] running on GCE node
W20260929 18:34:40.804981 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:40.805785 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:40.806432 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:40.807621 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790706880807595 us; error 46 us; skew 500 ppm
Sep 29 18:34:40 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:40.810698 16889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:34:40.811578 16889 webserver.cc:460] Webserver started at http://127.0.98.1:35879/ using document root <none> and password file <none>
I20260929 18:34:40.812137 16889 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:40.812284 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:40.816222 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:34:40.818594 16937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:40.819916 16889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:34:40.820066 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "99dbc565a91342b1b1356f075a22d654"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "c8921f5866fefdd3614b492a64efb61c"
server_key_iv: "5ec170568fac390bc65601f1aa182ba7"
server_key_version: "encryptionkey@0"
I20260929 18:34:40.820678 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:40.834223 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:40.837456 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:40.837668 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:40.838238 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:40.839330 16944 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:34:40.845008 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:34:40.845104 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 18:34:40.845191 16889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:34:40.846916 16889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:34:40.846990 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:34:40.847292 16944 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Bootstrap starting.
I20260929 18:34:40.857788 16889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260929 18:34:40.857998 17051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260929 18:34:40.858784 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:34:40.864965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 16889
I20260929 18:34:40.865432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38457
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=43767
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:40.868945 16944 log.cc:824] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:40.877657 16944 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:40.878062 16944 tablet_bootstrap.cc:492] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Bootstrap complete.
I20260929 18:34:40.879488 16944 ts_tablet_manager.cc:1397] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260929 18:34:40.882735 16944 raft_consensus.cc:348] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:40.883250 16944 raft_consensus.cc:729] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dbc565a91342b1b1356f075a22d654, State: Initialized, Role: FOLLOWER
I20260929 18:34:40.883744 16944 consensus_queue.cc:260] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:40.884783 16944 ts_tablet_manager.cc:1428] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260929 18:34:40.903208 17054 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:34:40.860699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:59839)
0929 18:34:40.861184 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:40.861237 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:40.862029 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:40.902303 (+ 40274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":146,"threads_started":1}
W20260929 18:34:40.903868 17052 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40839 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40839: Client connection negotiation failed: client connection to 127.0.98.62:40839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:34:40.904663 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:40.904360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:51743)
0929 18:34:40.904490 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:40.904583 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:34:40.905342 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:40.905100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:48181)
0929 18:34:40.905241 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:40.905291 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 18:34:40.905542 17052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:34:40.976773 17055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:40.977054 17055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:40.977099 17055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:40.981308 17055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:40.981369 17055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:40.981457 17055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:34:40.987452 17055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43767
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:40.988622 17055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:40.989647 17055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:40.999298 17063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:40.999322 17062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:40.999294 17065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:40.999531 17055 server_base.cc:1034] running on GCE node
I20260929 18:34:41.000615 17055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:41.001240 17055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:41.002454 17055 hybrid_clock.cc:648] HybridClock initialized: now 1790706881002419 us; error 49 us; skew 500 ppm
Sep 29 18:34:41 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:41.005297 17055 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:34:41.006114 17055 webserver.cc:460] Webserver started at http://127.0.98.2:43767/ using document root <none> and password file <none>
I20260929 18:34:41.006659 17055 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:41.006716 17055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:41.010596 17055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:41.013032 17072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.014307 17055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:34:41.014441 17055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "986d1c9a9da34f109edd050299afdae5"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-fmv8"
server_key: "640a85a3e6e74067d8c08497bb2e41b9"
server_key_iv: "0f13f242bcede8654c19e3ec526f3a8b"
server_key_version: "encryptionkey@0"
I20260929 18:34:41.015058 17055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:41.027429 17055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:41.030812 17055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:41.031062 17055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:41.031706 17055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:41.032663 17055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:41.032737 17055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.032807 17055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:41.032850 17055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.042495 17055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38457
I20260929 18:34:41.042520 17185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38457 every 8 connection(s)
I20260929 18:34:41.043596 17055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:41.047461 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.045772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:58739)
0929 18:34:41.046212 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.046315 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
W20260929 18:34:41.047999 17186 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40839 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40839: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
I20260929 18:34:41.048600 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.048397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:52863)
0929 18:34:41.048493 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.048550 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:34:41.049189 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.049010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:59181)
0929 18:34:41.049102 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.049145 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:34:41.049355 17186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:34:41.052439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17055
I20260929 18:34:41.052796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42967
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40839
--webserver_port=35861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:41.161463 17189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:41.161751 17189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:41.161794 17189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:41.165678 17189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:41.165728 17189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:41.165784 17189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:34:41.170786 17189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35861
--tserver_master_addrs=127.0.98.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:41.171938 17189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:41.173069 17189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:34:41.184046 17189 server_base.cc:1034] running on GCE node
W20260929 18:34:41.184065 17197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:41.184360 17195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:41.184377 17194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:41.184871 17189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:41.185621 17189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:41.186860 17189 hybrid_clock.cc:648] HybridClock initialized: now 1790706881186815 us; error 50 us; skew 500 ppm
Sep 29 18:34:41 dist-test-slave-fmv8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:41.189967 17189 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:34:41.190925 17189 webserver.cc:460] Webserver started at http://127.0.98.3:35861/ using document root <none> and password file <none>
I20260929 18:34:41.191504 17189 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:41.191568 17189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:41.196620 17189 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:34:41.198958 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.200412 17189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:41.200531 17189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "569d50f3c7ce4b6181c370d1ec96ecf5"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-fmv8"
server_key: "94c0903d1300ce08e32b13d65a73aa25"
server_key_iv: "c126b0d852b34fd497c97972dece970e"
server_key_version: "encryptionkey@0"
I20260929 18:34:41.201097 17189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:41.213446 17189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:41.216717 17189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:41.216871 17189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:41.217412 17189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:41.218353 17189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:41.218405 17189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.218444 17189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:41.218461 17189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:41.228442 17189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42967
I20260929 18:34:41.228468 17317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42967 every 8 connection(s)
I20260929 18:34:41.229602 17189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:41.233739 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.231906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:50187)
0929 18:34:41.232430 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.232545 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":118,"threads_started":1}
W20260929 18:34:41.234227 17318 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40839 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40839: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
I20260929 18:34:41.234860 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.234618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:34191)
0929 18:34:41.234750 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.234802 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:34:41.235523 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.235325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:52259)
0929 18:34:41.235428 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.235474 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:34:41.235780 17318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:34:41.239223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17189
I20260929 18:34:41.906836 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:41.906200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:37365)
0929 18:34:41.906649 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:41.906719 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260929 18:34:42.020624 17323 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:42.020802 17323 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:42.021903 17323 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
Sep 29 18:34:42 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 18:34:42 dist-test-slave-fmv8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 18:34:42.040333 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.022214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38457 (local address 127.0.98.1:45857)
0929 18:34:42.022342 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:42.022358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:42.022459 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:42.024184 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:42.024191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:42.024738 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:42.025206 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:42.025221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.026472 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.026476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:42.027311 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:42.027319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.028044 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.028048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:42.028270 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:42.033509 (+ 5239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:42.033546 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:42.035651 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:42.037961 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:42.037963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:42.037965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:42.038277 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:42.038707 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:42.038714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:42.038808 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:42.039398 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:42.039400 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:42.039602 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 18:34:42.039760 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 18:34:42.040123 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":128,"spinlock_wait_cycles":4480}
I20260929 18:34:42.040345 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.022336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41271 (local address 127.0.98.3:42967)
0929 18:34:42.022803 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:42.022818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:42.023000 (+ 182us) server_negotiation.cc:407] Connection header received
0929 18:34:42.023813 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:42.023819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:42.024209 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:42.024511 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:42.025380 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.026323 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.027439 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.027889 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.028071 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:42.035604 (+ 7533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:42.035612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:42.035627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:42.035658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:42.037522 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038187 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:42.038196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:42.038203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:42.038362 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038730 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:42.038733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:42.038735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:42.039184 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:42.039284 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:42.039867 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 18:34:42.040125 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":88,"threads_started":1}
I20260929 18:34:42.040350 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.022326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45857 (local address 127.0.98.2:38457)
0929 18:34:42.022683 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:42.022696 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:42.022731 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:34:42.023388 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:42.023417 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:42.023799 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:42.024098 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:42.025375 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.026328 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.027433 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.027919 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.028078 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:42.035846 (+ 7768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:42.035851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:42.035861 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:42.035888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:42.037839 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038392 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:42.038397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:42.038401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:42.038589 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038899 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:42.038901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:42.038902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:42.039293 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:42.039417 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:42.039867 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:34:42.040130 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260929 18:34:42.040392 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.022218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.98.1:41271)
0929 18:34:42.022892 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:42.022904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:42.023026 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:42.024549 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:42.024552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:42.024803 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:42.025206 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:42.025221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.026471 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.026475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:42.027311 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:42.027320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:42.028018 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:42.028021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:42.028261 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:42.033517 (+ 5256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:42.033539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:42.035326 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:42.037650 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:42.037658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:42.037675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:42.038054 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:42.038470 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:42.038476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:42.038478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:42.038637 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:42.039301 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:42.039307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:42.039575 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 18:34:42.039760 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 18:34:42.040232 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":649,"spinlock_wait_cycles":26752,"thread_start_us":72,"threads_started":1}
W20260929 18:34:42.043599 16939 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
W20260929 18:34:42.043828 16940 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:42.043920 16940 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99dbc565a91342b1b1356f075a22d654; no voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5
I20260929 18:34:42.044245 17323 raft_consensus.cc:2726] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:34:42.050189 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.049860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:40069)
0929 18:34:42.050068 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:42.050116 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260929 18:34:42.236866 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.236419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:41529)
0929 18:34:42.236715 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:42.236774 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260929 18:34:42.908129 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:42.907686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:50853)
0929 18:34:42.907965 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:42.908030 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260929 18:34:43.051321 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:43.050879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:55299)
0929 18:34:43.051167 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:43.051223 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260929 18:34:43.238122 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:43.237694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:39581)
0929 18:34:43.237957 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:43.238019 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260929 18:34:43.909509 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:43.909034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:43869)
0929 18:34:43.909332 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:43.909396 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260929 18:34:44.052510 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:44.052055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:58263)
0929 18:34:44.052355 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:44.052420 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":151,"threads_started":1}
I20260929 18:34:44.070823 17334 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:44.070976 17334 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:44.071542 17334 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
W20260929 18:34:44.072415 16939 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
W20260929 18:34:44.072566 16940 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:44.072662 16940 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99dbc565a91342b1b1356f075a22d654; no voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5
I20260929 18:34:44.072844 17334 raft_consensus.cc:2726] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:34:44.239396 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:44.238976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:59465)
0929 18:34:44.239234 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:44.239287 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260929 18:34:44.910774 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:44.910285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:41995)
0929 18:34:44.910564 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:44.910668 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
I20260929 18:34:45.053779 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:45.053345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:51603)
0929 18:34:45.053628 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:45.053688 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260929 18:34:45.240700 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:45.240305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:33595)
0929 18:34:45.240551 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:45.240609 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260929 18:34:45.842577 17339 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:45.842775 17339 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:45.843158 17339 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
W20260929 18:34:45.843968 16939 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
W20260929 18:34:45.844074 16940 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:45.844201 16940 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99dbc565a91342b1b1356f075a22d654; no voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5
I20260929 18:34:45.844357 17339 raft_consensus.cc:2726] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:34:45.912346 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:45.911769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:60361)
0929 18:34:45.912112 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:45.912235 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":77,"threads_started":1}
I20260929 18:34:46.055039 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:46.054604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:48791)
0929 18:34:46.054884 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:46.054945 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260929 18:34:46.241998 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:46.241542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:39691)
0929 18:34:46.241847 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:46.241906 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260929 18:34:46.913691 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:46.913221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:47907)
0929 18:34:46.913532 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:46.913596 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
I20260929 18:34:47.056494 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:47.055888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:55129)
0929 18:34:47.056337 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:47.056399 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260929 18:34:47.243326 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:47.242847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:53373)
0929 18:34:47.243167 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:47.243230 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260929 18:34:47.915112 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:47.914631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:59435)
0929 18:34:47.914958 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:47.915009 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260929 18:34:48.058000 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:48.057348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:53253)
0929 18:34:48.057783 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:48.057853 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":98,"threads_started":1}
I20260929 18:34:48.244959 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:48.244303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:34601)
0929 18:34:48.244727 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:48.244784 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260929 18:34:48.916257 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:48.915828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:37647)
0929 18:34:48.916079 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:48.916134 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260929 18:34:49.059126 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:49.058743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:45283)
0929 18:34:49.058980 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:49.059034 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260929 18:34:49.246263 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:49.245799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:46629)
0929 18:34:49.246057 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:49.246111 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260929 18:34:49.632858 17352 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:49.633008 17352 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:49.633489 17352 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
W20260929 18:34:49.634268 16939 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
W20260929 18:34:49.634392 16940 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:49.634488 16940 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99dbc565a91342b1b1356f075a22d654; no voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5
I20260929 18:34:49.634660 17352 raft_consensus.cc:2726] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:34:49.917492 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:49.917070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:38619)
0929 18:34:49.917330 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:49.917394 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260929 18:34:50.060488 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:50.059948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:45793)
0929 18:34:50.060323 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:50.060388 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260929 18:34:50.247611 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:50.247097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:50925)
0929 18:34:50.247442 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:50.247511 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":79,"threads_started":1}
I20260929 18:34:50.918802 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:50.918400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:41387)
0929 18:34:50.918646 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:50.918703 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260929 18:34:51.061661 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:51.061247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:45437)
0929 18:34:51.061509 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:51.061566 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260929 18:34:51.248939 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:51.248508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:46449)
0929 18:34:51.248784 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:51.248845 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260929 18:34:51.920030 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:51.919619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:46355)
0929 18:34:51.919868 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:51.919929 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260929 18:34:52.050871 17360 raft_consensus.cc:482] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:52.051002 17360 raft_consensus.cc:504] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569d50f3c7ce4b6181c370d1ec96ecf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "99dbc565a91342b1b1356f075a22d654" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "986d1c9a9da34f109edd050299afdae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38457 } }
I20260929 18:34:52.051548 17360 leader_election.cc:288] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967), 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457)
W20260929 18:34:52.052430 16939 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 569d50f3c7ce4b6181c370d1ec96ecf5 (127.0.98.3:42967): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
W20260929 18:34:52.052749 16940 leader_election.cc:341] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 986d1c9a9da34f109edd050299afdae5 (127.0.98.2:38457): Not found: Tablet not found: 09d9a59c4e8b40028dbe240183499dba
I20260929 18:34:52.052839 16940 leader_election.cc:302] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99dbc565a91342b1b1356f075a22d654; no voters: 569d50f3c7ce4b6181c370d1ec96ecf5, 986d1c9a9da34f109edd050299afdae5
I20260929 18:34:52.052991 17360 raft_consensus.cc:2726] T 09d9a59c4e8b40028dbe240183499dba P 99dbc565a91342b1b1356f075a22d654 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:34:52.063135 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:52.062601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:58299)
0929 18:34:52.062926 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:52.062998 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260929 18:34:52.250173 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:52.249731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:55477)
0929 18:34:52.249996 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:52.250053 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260929 18:34:52.921314 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:52.920850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:45167)
0929 18:34:52.921125 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:52.921216 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260929 18:34:53.064448 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:53.063970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:52571)
0929 18:34:53.064293 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:53.064353 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
I20260929 18:34:53.251467 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:53.251002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:38679)
0929 18:34:53.251289 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:53.251355 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260929 18:34:53.922601 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:53.922143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:49043)
0929 18:34:53.922420 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:53.922477 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260929 18:34:54.065644 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.065212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:39583)
0929 18:34:54.065493 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:54.065548 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260929 18:34:54.252799 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.252348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:41117)
0929 18:34:54.252640 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:54.252696 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260929 18:34:54.923763 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.923339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:52897)
0929 18:34:54.923601 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:54.923661 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260929 18:34:55.066855 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.066450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:58013)
0929 18:34:55.066701 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.066761 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260929 18:34:55.254123 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.253684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.3:58979)
0929 18:34:55.253977 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.254033 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260929 18:34:55.925308 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.924851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.1:49657)
0929 18:34:55.925116 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.925204 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260929 18:34:56.068361 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.067892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40839 (local address 127.0.98.2:36207)
0929 18:34:56.068189 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.068261 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260929 18:34:56.241137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 16889
I20260929 18:34:56.246994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17055
I20260929 18:34:56.252413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17189
2026-09-29T18:34:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59092 ms)
[----------] 30 tests from SecurityITest (240470 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:34:56 dist-test-slave-fmv8 krb5kdc[17376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:34:56 dist-test-slave-fmv8 krb5kdc[17376](info): set up 2 sockets
Sep 29 18:34:56 dist-test-slave-fmv8 krb5kdc[17376](info): commencing operation
krb5kdc: starting...
W20260929 18:34:57.320012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:34:57 dist-test-slave-fmv8 krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:34:57.441283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:57.552207 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:57.552541 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:57.552621 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:57.556792 17389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:57.556883 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:57.556922 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:57.556980 17389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:57.557017 17389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:57.562340 17389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:57.563437 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:57.564543 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:57.573307 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:57.573328 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:57.573457 17389 server_base.cc:1034] running on GCE node
W20260929 18:34:57.573553 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:57.574257 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:57.575354 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:57.576517 17389 hybrid_clock.cc:648] HybridClock initialized: now 1790706897576497 us; error 34 us; skew 500 ppm
Sep 29 18:34:57 dist-test-slave-fmv8 krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:57.579346 17389 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:34:57.580298 17389 webserver.cc:460] Webserver started at http://127.0.98.62:41429/ using document root <none> and password file <none>
I20260929 18:34:57.580850 17389 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:57.580909 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:57.581142 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:57.583142 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "75f8f72a57e547e39f658c42c7e42e0c"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "73ecc356b13df0903ff9dad21be7a5d9"
server_key_iv: "937359fb3766683badf35ab6d6b91035"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.583689 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "75f8f72a57e547e39f658c42c7e42e0c"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "73ecc356b13df0903ff9dad21be7a5d9"
server_key_iv: "937359fb3766683badf35ab6d6b91035"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.587409 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:34:57.589830 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:57.590999 17389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:34:57.591136 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "75f8f72a57e547e39f658c42c7e42e0c"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "73ecc356b13df0903ff9dad21be7a5d9"
server_key_iv: "937359fb3766683badf35ab6d6b91035"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.591243 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:57.617726 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:57.621176 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:57.621392 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:57.629051 17389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43043
I20260929 18:34:57.629081 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43043 every 8 connection(s)
I20260929 18:34:57.630021 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:57.632781 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:57.637638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17389
I20260929 18:34:57.637779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:34:57.638054 392 external_mini_cluster.cc:1428] Setting key 59c6e97c9b17daba15d3f0f831cd8ff3
I20260929 18:34:57.638336 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: Bootstrap starting.
I20260929 18:34:57.640895 17457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:57.641673 17457 log.cc:824] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:57.643517 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: No bootstrap required, opened a new log
Sep 29 18:34:57 dist-test-slave-fmv8 krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:57.647034 17457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260929 18:34:57.647272 17457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:57.647336 17457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f8f72a57e547e39f658c42c7e42e0c, State: Initialized, Role: FOLLOWER
I20260929 18:34:57.647785 17457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260929 18:34:57.647933 17457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:57.648000 17457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:57.648171 17457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:57.649029 17457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260929 18:34:57.649384 17457 leader_election.cc:302] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f8f72a57e547e39f658c42c7e42e0c; no voters:
I20260929 18:34:57.649711 17457 leader_election.cc:288] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:34:57.649801 17462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:57.650017 17462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [term 1 LEADER]: Becoming Leader. State: Replica: 75f8f72a57e547e39f658c42c7e42e0c, State: Running, Role: LEADER
I20260929 18:34:57.650360 17462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260929 18:34:57.651026 17457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:57.652186 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f8f72a57e547e39f658c42c7e42e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260929 18:34:57.652199 17464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f8f72a57e547e39f658c42c7e42e0c. Latest consensus state: current_term: 1 leader_uuid: "75f8f72a57e547e39f658c42c7e42e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f8f72a57e547e39f658c42c7e42e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260929 18:34:57.652427 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:57.652742 17464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:57.653039 17471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:34:57.653095 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:57.639395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.98.62:43043)
0929 18:34:57.639864 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:57.639879 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:57.639914 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:34:57.640633 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:57.640655 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:57.641022 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:57.641384 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:57.642180 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.643201 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.644281 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.644751 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.644922 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:57.647344 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:57.647351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:57.647370 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:57.647402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:57.649365 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:57.649868 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:57.649877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:57.649884 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:57.650042 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:57.650370 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:57.650373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:57.650374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:57.650824 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:57.651045 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:57.651645 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 18:34:57.651958 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":163,"threads_started":1}
I20260929 18:34:57.656174 17471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:34:57.661625 17471 catalog_manager.cc:1347] Generated new cluster ID: 19ece6527d9b444d991b7dd4cb207ec7
I20260929 18:34:57.661696 17471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:34:57.667959 17471 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:34:57.669224 17471 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:34:57.682462 17471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: Generated new TSK 0
I20260929 18:34:57.683200 17471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:34:57.718952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:57.831432 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:57.831686 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:57.831727 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:57.835740 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:57.835798 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:57.835856 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:34:57.840958 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:57.841943 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:57.842886 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:57.852680 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:57.852680 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:57.853055 17483 server_base.cc:1034] running on GCE node
W20260929 18:34:57.852768 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:57.853559 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:57.854161 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:57.855351 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790706897855319 us; error 42 us; skew 500 ppm
Sep 29 18:34:57 dist-test-slave-fmv8 krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:57.858147 17483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:34:57.858990 17483 webserver.cc:460] Webserver started at http://127.0.98.1:40779/ using document root <none> and password file <none>
I20260929 18:34:57.859493 17483 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:57.859550 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:57.859710 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:57.861757 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "b0b640266ea74c9d92aad57f7ccef104"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "7825605363476b61ecab000f85525176"
server_key_iv: "b0e6def9b3e7a01aeecbcd35b539937b"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.862265 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b0b640266ea74c9d92aad57f7ccef104"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "7825605363476b61ecab000f85525176"
server_key_iv: "b0e6def9b3e7a01aeecbcd35b539937b"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.865979 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:34:57.868462 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:57.869722 17483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:34:57.869843 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "b0b640266ea74c9d92aad57f7ccef104"
format_stamp: "Formatted at 2026-09-29 18:34:57 on dist-test-slave-fmv8"
server_key: "7825605363476b61ecab000f85525176"
server_key_iv: "b0e6def9b3e7a01aeecbcd35b539937b"
server_key_version: "encryptionkey@0"
I20260929 18:34:57.869949 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:57.889292 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:57.892706 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:57.892912 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:57.893555 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:57.894393 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:57.894441 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:57.894479 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:57.894541 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:57.904234 17483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37169
I20260929 18:34:57.904343 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37169 every 8 connection(s)
I20260929 18:34:57.905272 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:34:57 dist-test-slave-fmv8 krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:57.915622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17483
I20260929 18:34:57.915772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:34:57.916077 392 external_mini_cluster.cc:1428] Setting key 520f4a79496d414bc6812a25af787b5c
I20260929 18:34:57.920296 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:57.907251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56305 (local address 127.0.98.62:43043)
0929 18:34:57.907387 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:57.907391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:57.908337 (+ 946us) server_negotiation.cc:407] Connection header received
0929 18:34:57.909705 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:57.909708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:57.909764 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:57.909847 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:57.910986 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.911540 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.912768 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.913047 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.913164 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:57.915921 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:57.915924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:57.915927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:57.915952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:57.917772 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:57.918409 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:57.918412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:57.918419 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:57.918504 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:57.918925 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:57.918927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:57.918928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:57.919118 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:57.919196 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:57.919881 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 18:34:57.920061 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:34:57.921125 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:57.907616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43043 (local address 127.0.98.1:56305)
0929 18:34:57.908095 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:57.908215 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:57.909495 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:57.910027 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:57.910033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:57.910384 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:57.910842 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:57.910854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.911667 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.911670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:57.912643 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:57.912652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:57.913202 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:57.913206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:57.913371 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:57.913989 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:57.914006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:57.915671 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:57.917882 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:57.917888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:57.917934 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:57.918293 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:57.918610 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:57.918616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:57.918621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:57.918820 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:57.919225 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:57.919234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:57.919581 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 18:34:57.919800 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 18:34:57.920195 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260929 18:34:57.922437 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43043
I20260929 18:34:57.922704 17612 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:57.923316 17612 heartbeater.cc:511] Master 127.0.98.62:43043 requested a full tablet report, sending...
I20260929 18:34:57.924957 17421 ts_manager.cc:194] Registered new tserver with Master: b0b640266ea74c9d92aad57f7ccef104 (127.0.98.1:37169)
I20260929 18:34:57.926082 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56305
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:34:57.944017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:58.057716 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:58.057968 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:58.058012 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:58.062086 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:58.062150 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:58.062239 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:34:58.067502 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:58.068665 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:58.069690 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:58.079742 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:58.079764 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.079831 17617 server_base.cc:1034] running on GCE node
W20260929 18:34:58.079766 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.080507 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:58.081184 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:58.082366 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790706898082350 us; error 30 us; skew 500 ppm
Sep 29 18:34:58 dist-test-slave-fmv8 krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:58.085408 17617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:34:58.086261 17617 webserver.cc:460] Webserver started at http://127.0.98.2:36193/ using document root <none> and password file <none>
I20260929 18:34:58.086808 17617 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:58.086870 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:58.087074 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:58.089098 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "48a170d58f164b319dc5ac517cc098cb"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "e26c038f2934962218f1725eb1b45846"
server_key_iv: "fef3b954c1cbb622d76c7f947b453d20"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.089709 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48a170d58f164b319dc5ac517cc098cb"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "e26c038f2934962218f1725eb1b45846"
server_key_iv: "fef3b954c1cbb622d76c7f947b453d20"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.093299 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 18:34:58.095743 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.097072 17617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:58.097198 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "48a170d58f164b319dc5ac517cc098cb"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "e26c038f2934962218f1725eb1b45846"
server_key_iv: "fef3b954c1cbb622d76c7f947b453d20"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.097301 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:58.123769 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:58.126984 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:58.127206 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:58.127828 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:58.128738 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:58.128813 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.128883 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:58.128937 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.137738 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45083
I20260929 18:34:58.137809 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45083 every 8 connection(s)
I20260929 18:34:58.138656 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:58.140493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17617
I20260929 18:34:58.140622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:34:58.140918 392 external_mini_cluster.cc:1428] Setting key c84629a5031ebc0832db58749b9e726c
Sep 29 18:34:58 dist-test-slave-fmv8 krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:34:58.154731 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.140491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51831 (local address 127.0.98.62:43043)
0929 18:34:58.140658 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.140662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.141741 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 18:34:58.143157 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.143160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.143212 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.143291 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:58.144797 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.145330 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.147134 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.147345 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.147428 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.150473 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:58.150479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:58.150484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:58.150529 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:58.152331 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.152888 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.152891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.152893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.152989 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.153424 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.153430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.153433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.153689 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:58.153770 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.154306 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.154448 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:34:58.155323 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.140981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43043 (local address 127.0.98.2:51831)
0929 18:34:58.141590 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.141635 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.142932 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.143457 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.143464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.143910 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:58.144642 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.144663 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.145454 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.145461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.147003 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.147015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.147517 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.147525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.147771 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.148667 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:58.148688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:58.150306 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:58.152434 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:58.152439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:58.152450 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:58.152744 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:58.153099 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:58.153102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:58.153104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:58.153266 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:58.153801 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:58.153809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:58.154064 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.154221 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.154464 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":147,"threads_started":1}
I20260929 18:34:58.156574 17746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43043
I20260929 18:34:58.156818 17746 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:58.157387 17746 heartbeater.cc:511] Master 127.0.98.62:43043 requested a full tablet report, sending...
I20260929 18:34:58.158551 17421 ts_manager.cc:194] Registered new tserver with Master: 48a170d58f164b319dc5ac517cc098cb (127.0.98.2:45083)
I20260929 18:34:58.159126 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:34:58.171315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:58.282267 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:58.282523 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:58.282565 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:58.286603 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:58.286665 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:58.286722 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:34:58.291967 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:58.293076 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:58.294122 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:58.304178 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.304220 17751 server_base.cc:1034] running on GCE node
W20260929 18:34:58.304185 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:58.304184 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.304912 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:58.305586 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:58.306798 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790706898306778 us; error 32 us; skew 500 ppm
Sep 29 18:34:58 dist-test-slave-fmv8 krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:58.310122 17751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:34:58.310982 17751 webserver.cc:460] Webserver started at http://127.0.98.3:43935/ using document root <none> and password file <none>
I20260929 18:34:58.311563 17751 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:58.311654 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:58.311859 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:58.313920 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "060a443bbcb44a6aae6347f718dee1a1"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "6b005f0565ab3f71045d0be5931d6797"
server_key_iv: "16878d0765da2dad4dc17ccab5395cd5"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.314476 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "060a443bbcb44a6aae6347f718dee1a1"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "6b005f0565ab3f71045d0be5931d6797"
server_key_iv: "16878d0765da2dad4dc17ccab5395cd5"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.318202 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:34:58.320663 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.321945 17751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:34:58.322105 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "060a443bbcb44a6aae6347f718dee1a1"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "6b005f0565ab3f71045d0be5931d6797"
server_key_iv: "16878d0765da2dad4dc17ccab5395cd5"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.322237 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:58.333101 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:58.336367 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:58.336568 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:58.337134 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:58.337980 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:58.338052 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.338121 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:58.338165 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.346890 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40113
I20260929 18:34:58.346951 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40113 every 8 connection(s)
I20260929 18:34:58.347877 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:34:58 dist-test-slave-fmv8 krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:34:58.357954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17751
I20260929 18:34:58.358115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:34:58.358537 392 external_mini_cluster.cc:1428] Setting key 412a752f4f81155b2e7721cfb9374dbd
I20260929 18:34:58.361898 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.349713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43367 (local address 127.0.98.62:43043)
0929 18:34:58.349992 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.349997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.350821 (+ 824us) server_negotiation.cc:407] Connection header received
0929 18:34:58.352300 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.352303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.352353 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.352437 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:58.353641 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.354200 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.355292 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.355528 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.355613 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.358005 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:58.358009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:58.358012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:58.358046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:58.359750 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.360398 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.360402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.360404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.360499 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.360904 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.360907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.360909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.361138 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:58.361217 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.361688 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.361783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260929 18:34:58.362783 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.350196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43043 (local address 127.0.98.3:43367)
0929 18:34:58.350679 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.350721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.352059 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.352621 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.352631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.353009 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:58.353475 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.353490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.354310 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.354313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.355176 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.355184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.355657 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.355659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.355792 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.356414 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:58.356429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:58.357839 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:58.359899 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:58.359905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:58.359919 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:58.360298 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:58.360610 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:58.360612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:58.360614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:58.360788 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:58.361234 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:58.361239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:58.361475 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.361626 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.361861 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20260929 18:34:58.363957 17880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43043
I20260929 18:34:58.364215 17880 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:58.364658 17880 heartbeater.cc:511] Master 127.0.98.62:43043 requested a full tablet report, sending...
I20260929 18:34:58.365639 17421 ts_manager.cc:194] Registered new tserver with Master: 060a443bbcb44a6aae6347f718dee1a1 (127.0.98.3:40113)
I20260929 18:34:58.366145 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43367
I20260929 18:34:58.373121 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:58.382349 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.375185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.98.62:43043)
0929 18:34:58.375321 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.375326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.375548 (+ 222us) server_negotiation.cc:407] Connection header received
0929 18:34:58.375643 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.375646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.375687 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.375752 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:58.376477 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.376990 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.377852 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.378072 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.378161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.379522 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:58.379525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:58.379527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:58.379549 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:58.380838 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.381205 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.381208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.381209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.381281 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:58.381546 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:58.381549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:58.381550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:58.381746 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:58.381844 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.382050 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.382142 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:34:58.385134 17421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:34:58.388221 17421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:34:58.402244 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.396465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45083 (local address 127.0.0.1:50398)
0929 18:34:58.397100 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.397149 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.397269 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.397914 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.397920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.397941 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:58.398086 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.398099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.399319 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.399325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.400706 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.400716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.401742 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.401746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.401850 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.401886 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.402012 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.402131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":28,"thread_start_us":76,"threads_started":1}
I20260929 18:34:58.402639 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.397197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37169 (local address 127.0.0.1:59556)
0929 18:34:58.397596 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.397610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.397683 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.398316 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.398319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.398337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:58.398411 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.398416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.399780 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.399784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.400961 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.400968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.402254 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.402258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.402347 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.402368 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.402443 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.402508 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":90,"threads_started":1}
I20260929 18:34:58.402710 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.396609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.0.98.2:45083)
0929 18:34:58.397231 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.397243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.397268 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:34:58.397478 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.397527 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.397750 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.397917 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:58.398211 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.399201 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.400913 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.401614 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.401732 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.401924 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.402450 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.402555 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":110,"threads_started":1}
I20260929 18:34:58.403076 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.397473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.0.98.1:37169)
0929 18:34:58.397851 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.397868 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.397891 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:34:58.397969 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.397976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.398174 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.398336 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:58.398500 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.399635 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.401082 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.401907 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.402065 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.402202 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.402817 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.402926 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260929 18:34:58.403411 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.396584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.0.1:33626)
0929 18:34:58.397414 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.397427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.398147 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.398844 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.398847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.398864 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:58.398935 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.398940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.400441 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.400445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.401701 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.401712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.402968 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.402972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.403063 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.403088 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.403176 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.403247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":41,"thread_start_us":115,"threads_started":1}
I20260929 18:34:58.403473 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.396962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.0.98.3:40113)
0929 18:34:58.398408 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.398419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.398439 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:34:58.398510 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.398515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.398702 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.398849 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:58.399034 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.400214 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.401847 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.402660 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.402770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.402907 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.403228 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.403316 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":160,"threads_started":1}
I20260929 18:34:58.405746 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 24b686dc5c2d464181f380924051532e (DEFAULT_TABLE table=test-table [id=c3a1cb085b13400a9c2b0f5ffac21542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:58.405709 17680 tablet_service.cc:1467] Processing CreateTablet for tablet 24b686dc5c2d464181f380924051532e (DEFAULT_TABLE table=test-table [id=c3a1cb085b13400a9c2b0f5ffac21542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:58.405821 17814 tablet_service.cc:1467] Processing CreateTablet for tablet 24b686dc5c2d464181f380924051532e (DEFAULT_TABLE table=test-table [id=c3a1cb085b13400a9c2b0f5ffac21542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:58.406723 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b686dc5c2d464181f380924051532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:58.406737 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b686dc5c2d464181f380924051532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:58.406737 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b686dc5c2d464181f380924051532e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:58.413296 17897 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Bootstrap starting.
I20260929 18:34:58.413723 17899 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Bootstrap starting.
I20260929 18:34:58.415704 17897 tablet_bootstrap.cc:654] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:58.415704 17899 tablet_bootstrap.cc:654] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:58.416442 17898 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Bootstrap starting.
I20260929 18:34:58.417117 17899 log.cc:824] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:58.417117 17897 log.cc:824] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:58.418403 17898 tablet_bootstrap.cc:654] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:58.419215 17899 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: No bootstrap required, opened a new log
I20260929 18:34:58.419215 17897 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: No bootstrap required, opened a new log
I20260929 18:34:58.419404 17899 ts_tablet_manager.cc:1397] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:34:58.419404 17897 ts_tablet_manager.cc:1397] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:34:58.419538 17898 log.cc:824] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:58.421137 17898 tablet_bootstrap.cc:492] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: No bootstrap required, opened a new log
I20260929 18:34:58.421329 17898 ts_tablet_manager.cc:1397] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:34:58.423306 17899 raft_consensus.cc:348] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.423306 17897 raft_consensus.cc:348] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.423578 17899 raft_consensus.cc:374] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:58.423578 17897 raft_consensus.cc:374] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:58.423637 17897 raft_consensus.cc:729] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b640266ea74c9d92aad57f7ccef104, State: Initialized, Role: FOLLOWER
I20260929 18:34:58.423666 17899 raft_consensus.cc:729] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a170d58f164b319dc5ac517cc098cb, State: Initialized, Role: FOLLOWER
I20260929 18:34:58.424165 17897 consensus_queue.cc:260] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.424165 17899 consensus_queue.cc:260] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.424257 17898 raft_consensus.cc:348] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.424575 17898 raft_consensus.cc:374] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:58.424650 17898 raft_consensus.cc:729] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060a443bbcb44a6aae6347f718dee1a1, State: Initialized, Role: FOLLOWER
I20260929 18:34:58.425076 17898 consensus_queue.cc:260] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.425153 17899 ts_tablet_manager.cc:1428] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:34:58.425153 17897 ts_tablet_manager.cc:1428] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:34:58.425489 17746 heartbeater.cc:503] Master 127.0.98.62:43043 was elected leader, sending a full tablet report...
I20260929 18:34:58.425789 17612 heartbeater.cc:503] Master 127.0.98.62:43043 was elected leader, sending a full tablet report...
I20260929 18:34:58.426118 17898 ts_tablet_manager.cc:1428] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:34:58.426406 17880 heartbeater.cc:503] Master 127.0.98.62:43043 was elected leader, sending a full tablet report...
W20260929 18:34:58.598850 17881 tablet.cc:2369] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:34:58.639818 17747 tablet.cc:2369] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:34:58.656778 17613 tablet.cc:2369] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:58.718103 17904 raft_consensus.cc:482] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:58.718312 17904 raft_consensus.cc:504] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.719411 17904 leader_election.cc:288] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060a443bbcb44a6aae6347f718dee1a1 (127.0.98.3:40113), b0b640266ea74c9d92aad57f7ccef104 (127.0.98.1:37169)
I20260929 18:34:58.723472 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.719917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58773 (local address 127.0.98.3:40113)
0929 18:34:58.720084 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.720090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.720172 (+ 82us) server_negotiation.cc:407] Connection header received
0929 18:34:58.720345 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.720348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.720400 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.720491 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:58.720750 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.721265 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.722473 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.722905 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.723028 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.723104 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.723275 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.723342 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:34:58.723495 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.719693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.98.2:58773)
0929 18:34:58.720064 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.720081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.720237 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.720501 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.720504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.720519 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:58.720643 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.720648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.721535 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.721541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.722341 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.722349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.723035 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.723039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.723140 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.723171 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.723259 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.723342 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":6,"thread_start_us":73,"threads_started":1}
I20260929 18:34:58.723678 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.719693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37169 (local address 127.0.98.2:52853)
0929 18:34:58.720126 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.720138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:58.720323 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:58.720590 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:58.720593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:58.720608 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:58.720742 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.720748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.721473 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.721478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.722352 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:58.722361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.723219 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.723222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:58.723350 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.723375 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:34:58.723468 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:34:58.723538 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":157,"thread_start_us":63,"threads_started":1}
I20260929 18:34:58.723695 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.719842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52853 (local address 127.0.98.1:37169)
0929 18:34:58.719997 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.720002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.720281 (+ 279us) server_negotiation.cc:407] Connection header received
0929 18:34:58.720435 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.720439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.720492 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.720578 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:58.720860 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.721358 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.722571 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.723068 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.723204 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.723264 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.723518 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.723589 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:34:58.724134 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b686dc5c2d464181f380924051532e" candidate_uuid: "48a170d58f164b319dc5ac517cc098cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060a443bbcb44a6aae6347f718dee1a1" is_pre_election: true
I20260929 18:34:58.724411 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b686dc5c2d464181f380924051532e" candidate_uuid: "48a170d58f164b319dc5ac517cc098cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b640266ea74c9d92aad57f7ccef104" is_pre_election: true
I20260929 18:34:58.724500 17834 raft_consensus.cc:2454] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48a170d58f164b319dc5ac517cc098cb in term 0.
I20260929 18:34:58.724694 17566 raft_consensus.cc:2454] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48a170d58f164b319dc5ac517cc098cb in term 0.
I20260929 18:34:58.725077 17634 leader_election.cc:302] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060a443bbcb44a6aae6347f718dee1a1, 48a170d58f164b319dc5ac517cc098cb; no voters:
I20260929 18:34:58.725337 17904 raft_consensus.cc:2781] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:34:58.725438 17904 raft_consensus.cc:482] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:34:58.725492 17904 raft_consensus.cc:3037] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:58.726622 17904 raft_consensus.cc:504] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.726960 17904 leader_election.cc:288] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [CANDIDATE]: Term 1 election: Requested vote from peers 060a443bbcb44a6aae6347f718dee1a1 (127.0.98.3:40113), b0b640266ea74c9d92aad57f7ccef104 (127.0.98.1:37169)
I20260929 18:34:58.727319 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b686dc5c2d464181f380924051532e" candidate_uuid: "48a170d58f164b319dc5ac517cc098cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060a443bbcb44a6aae6347f718dee1a1"
I20260929 18:34:58.727485 17834 raft_consensus.cc:3037] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:58.727411 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b686dc5c2d464181f380924051532e" candidate_uuid: "48a170d58f164b319dc5ac517cc098cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b640266ea74c9d92aad57f7ccef104"
I20260929 18:34:58.727602 17566 raft_consensus.cc:3037] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:58.728637 17834 raft_consensus.cc:2454] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a170d58f164b319dc5ac517cc098cb in term 1.
I20260929 18:34:58.728766 17566 raft_consensus.cc:2454] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a170d58f164b319dc5ac517cc098cb in term 1.
I20260929 18:34:58.728969 17634 leader_election.cc:302] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060a443bbcb44a6aae6347f718dee1a1, 48a170d58f164b319dc5ac517cc098cb; no voters:
I20260929 18:34:58.729171 17904 raft_consensus.cc:2781] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:58.729337 17904 raft_consensus.cc:686] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 1 LEADER]: Becoming Leader. State: Replica: 48a170d58f164b319dc5ac517cc098cb, State: Running, Role: LEADER
I20260929 18:34:58.729674 17904 consensus_queue.cc:237] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } }
I20260929 18:34:58.733055 17419 catalog_manager.cc:5495] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 48a170d58f164b319dc5ac517cc098cb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "48a170d58f164b319dc5ac517cc098cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a170d58f164b319dc5ac517cc098cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 } health_report { overall_health: UNKNOWN } } }
I20260929 18:34:58.764791 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.761194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.98.2:45083)
0929 18:34:58.761326 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:58.761330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:58.761343 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:34:58.761424 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:58.761426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:58.761478 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:58.761572 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:34:58.761822 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.762325 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.763293 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:58.763522 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:58.763605 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:58.763883 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:34:58.764331 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:34:58.764420 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:58.764553 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:34:58.764603 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:34:58.770750 17566 raft_consensus.cc:1261] T 24b686dc5c2d464181f380924051532e P b0b640266ea74c9d92aad57f7ccef104 [term 1 FOLLOWER]: Refusing update from remote peer 48a170d58f164b319dc5ac517cc098cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:58.770749 17834 raft_consensus.cc:1261] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 1 FOLLOWER]: Refusing update from remote peer 48a170d58f164b319dc5ac517cc098cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:58.771520 17904 consensus_queue.cc:1035] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "060a443bbcb44a6aae6347f718dee1a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:58.771736 17908 consensus_queue.cc:1035] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b640266ea74c9d92aad57f7ccef104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:58.781823 17912 mvcc.cc:205] Tried to move back new op lower bound from 7334735457358004224 to 7334735457199493120. Current Snapshot: MvccSnapshot[applied={T|T < 7334735457358004224}]
I20260929 18:34:58.787446 17419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:58.787628 17419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:58.790014 17419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75f8f72a57e547e39f658c42c7e42e0c: Sending DeleteTablet for 3 replicas of tablet 24b686dc5c2d464181f380924051532e
I20260929 18:34:58.790872 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b686dc5c2d464181f380924051532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33626
I20260929 18:34:58.791038 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b686dc5c2d464181f380924051532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50398
I20260929 18:34:58.791414 17921 tablet_replica.cc:330] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: stopping tablet replica
I20260929 18:34:58.791460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17483
I20260929 18:34:58.791487 17922 tablet_replica.cc:330] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: stopping tablet replica
I20260929 18:34:58.791517 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b686dc5c2d464181f380924051532e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59556
I20260929 18:34:58.791772 17921 raft_consensus.cc:2229] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:58.792128 17922 raft_consensus.cc:2229] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:34:58.792243 17921 raft_consensus.cc:2258] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:58.792488 17922 raft_consensus.cc:2258] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:58.793869 17921 ts_tablet_manager.cc:1905] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:58.795187 17922 ts_tablet_manager.cc:1905] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:58.796909 17921 ts_tablet_manager.cc:1918] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:58.797044 17921 log.cc:1196] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/24b686dc5c2d464181f380924051532e
I20260929 18:34:58.797446 17921 ts_tablet_manager.cc:1939] T 24b686dc5c2d464181f380924051532e P 060a443bbcb44a6aae6347f718dee1a1: Deleting consensus metadata
I20260929 18:34:58.798648 17922 ts_tablet_manager.cc:1918] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:58.798749 17922 log.cc:1196] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/24b686dc5c2d464181f380924051532e
I20260929 18:34:58.798766 17406 catalog_manager.cc:4841] TS 060a443bbcb44a6aae6347f718dee1a1 (127.0.98.3:40113): tablet 24b686dc5c2d464181f380924051532e (table test-table [id=c3a1cb085b13400a9c2b0f5ffac21542]) successfully deleted
I20260929 18:34:58.799137 17922 ts_tablet_manager.cc:1939] T 24b686dc5c2d464181f380924051532e P 48a170d58f164b319dc5ac517cc098cb: Deleting consensus metadata
W20260929 18:34:58.799108 17405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37169 (error 108)
I20260929 18:34:58.800123 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:58.799818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37169 (local address 127.0.0.1:59560)
0929 18:34:58.799963 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:58.800036 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:34:58.800206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17617
W20260929 18:34:58.801381 17405 catalog_manager.cc:4584] TS b0b640266ea74c9d92aad57f7ccef104 (127.0.98.1:37169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24b686dc5c2d464181f380924051532e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37169: connect: Connection refused (error 111)
I20260929 18:34:58.801599 17405 catalog_manager.cc:4841] TS 48a170d58f164b319dc5ac517cc098cb (127.0.98.2:45083): tablet 24b686dc5c2d464181f380924051532e (table test-table [id=c3a1cb085b13400a9c2b0f5ffac21542]) successfully deleted
I20260929 18:34:58.805202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17751
I20260929 18:34:58.810304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17389
2026-09-29T18:34:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2557 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:34:58Z Disabled control of system clock
I20260929 18:34:58.840903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46755
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:34:58.950230 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:58.950505 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:58.950541 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:58.954459 17927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:58.954511 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:58.954527 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:58.954543 17927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:58.954558 17927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:58.959263 17927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:58.960294 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:58.961609 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:58.970530 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.970597 17927 server_base.cc:1034] running on GCE node
W20260929 18:34:58.970523 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:58.970526 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:58.971482 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:58.972798 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:58.974032 17927 hybrid_clock.cc:648] HybridClock initialized: now 1790706898974013 us; error 33 us; skew 500 ppm
I20260929 18:34:58.976046 17927 webserver.cc:460] Webserver started at http://127.0.98.62:39101/ using document root <none> and password file <none>
I20260929 18:34:58.976689 17927 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:58.976780 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:58.977008 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:58.979097 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "693dac4f1f844e089b4f751836329526"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "50af62ea342cf0a9d15d81f81ca10bb6"
server_key_iv: "687eb7afc2295f1a21d0b6dd79567ad8"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.979657 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "693dac4f1f844e089b4f751836329526"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "50af62ea342cf0a9d15d81f81ca10bb6"
server_key_iv: "687eb7afc2295f1a21d0b6dd79567ad8"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.983314 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:34:58.985782 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:58.987083 17927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:34:58.987221 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "693dac4f1f844e089b4f751836329526"
format_stamp: "Formatted at 2026-09-29 18:34:58 on dist-test-slave-fmv8"
server_key: "50af62ea342cf0a9d15d81f81ca10bb6"
server_key_iv: "687eb7afc2295f1a21d0b6dd79567ad8"
server_key_version: "encryptionkey@0"
I20260929 18:34:58.987329 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:58.997916 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:58.998558 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:58.998735 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.005358 17927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46755
I20260929 18:34:59.005399 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46755 every 8 connection(s)
I20260929 18:34:59.006250 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:59.006479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 17927
I20260929 18:34:59.006603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:34:59.006961 392 external_mini_cluster.cc:1428] Setting key 7a8548c01e06da83fb77abd2368b219c
I20260929 18:34:59.009522 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.015605 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: Bootstrap starting.
I20260929 18:34:59.016189 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.008421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.0.98.62:46755)
0929 18:34:59.009015 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.009031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.009064 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:34:59.009781 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.009798 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.010174 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.010453 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.010778 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.011749 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.012978 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.013445 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.013565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.014383 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.014388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.014400 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:34:59.014769 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.014822 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:34:59.014841 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.014914 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.015076 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.015358 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260929 18:34:59.018049 17994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:59.018882 17994 log.cc:824] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:59.020701 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: No bootstrap required, opened a new log
I20260929 18:34:59.024046 17994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } }
I20260929 18:34:59.024307 17994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:59.024387 17994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693dac4f1f844e089b4f751836329526, State: Initialized, Role: FOLLOWER
I20260929 18:34:59.024788 17994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } }
I20260929 18:34:59.024920 17994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:59.024981 17994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:59.025096 17994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:59.025954 17994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } }
I20260929 18:34:59.026285 17994 leader_election.cc:302] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693dac4f1f844e089b4f751836329526; no voters:
I20260929 18:34:59.026568 17994 leader_election.cc:288] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:34:59.026685 17999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:59.026903 17999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [term 1 LEADER]: Becoming Leader. State: Replica: 693dac4f1f844e089b4f751836329526, State: Running, Role: LEADER
I20260929 18:34:59.027218 17999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } }
I20260929 18:34:59.027694 17994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:59.028787 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693dac4f1f844e089b4f751836329526. Latest consensus state: current_term: 1 leader_uuid: "693dac4f1f844e089b4f751836329526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } } }
I20260929 18:34:59.028914 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:59.029290 18008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:34:59.029927 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "693dac4f1f844e089b4f751836329526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693dac4f1f844e089b4f751836329526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46755 } } }
I20260929 18:34:59.030032 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:59.032370 18008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:34:59.037643 18008 catalog_manager.cc:1347] Generated new cluster ID: bec0237bc680444391b891d083bb292e
I20260929 18:34:59.037708 18008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:34:59.055922 18008 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:34:59.057240 18008 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:34:59.067895 18008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: Generated new TSK 0
I20260929 18:34:59.068758 18008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:34:59.079733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:34:59.195451 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.195710 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.195747 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.199680 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.199733 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.199787 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:34:59.204522 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:59.205555 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.206905 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.216488 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.216476 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.216476 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.216653 18018 server_base.cc:1034] running on GCE node
I20260929 18:34:59.217150 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.217815 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.219014 18018 hybrid_clock.cc:648] HybridClock initialized: now 1790706899218984 us; error 47 us; skew 500 ppm
I20260929 18:34:59.220883 18018 webserver.cc:460] Webserver started at http://127.0.98.1:45199/ using document root <none> and password file <none>
I20260929 18:34:59.221464 18018 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.221554 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.221777 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.223798 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "248b23fe9efd4fea805210b01ba81106"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "ab62549efad5f5c124e6d9dcc4bd7c1e"
server_key_iv: "007b91d65cf92886207b284f1b8f5eca"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.224397 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "248b23fe9efd4fea805210b01ba81106"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "ab62549efad5f5c124e6d9dcc4bd7c1e"
server_key_iv: "007b91d65cf92886207b284f1b8f5eca"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.228034 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:59.230747 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.232007 18018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:34:59.232177 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "248b23fe9efd4fea805210b01ba81106"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "ab62549efad5f5c124e6d9dcc4bd7c1e"
server_key_iv: "007b91d65cf92886207b284f1b8f5eca"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.232302 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:59.242452 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:59.243083 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:59.243263 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.243809 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:59.244767 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:59.244836 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.244900 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:59.244946 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.254035 18018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46499
I20260929 18:34:59.254065 18145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46499 every 8 connection(s)
I20260929 18:34:59.254891 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:34:59.256449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18018
I20260929 18:34:59.256569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:34:59.256863 392 external_mini_cluster.cc:1428] Setting key 81487eb4d0ffdfeb0eccf3f6ee975634
I20260929 18:34:59.258878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:34:59.265543 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.256628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51259 (local address 127.0.98.62:46755)
0929 18:34:59.257356 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.257363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.258039 (+ 676us) server_negotiation.cc:407] Connection header received
0929 18:34:59.259605 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.259612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.259678 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.259774 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.260920 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.261489 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.263346 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.263561 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.263637 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.265040 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.265043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.265045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:34:59.265059 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.265077 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:34:59.265084 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.265143 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.265327 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.265396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616}
I20260929 18:34:59.266849 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.257046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46755 (local address 127.0.98.1:51259)
0929 18:34:59.257888 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.257939 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.259385 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.259899 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.259905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.259991 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.260756 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.260778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.261627 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.261634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.263191 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.263206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.263712 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.263720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.263973 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.264690 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:34:59.264694 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.264759 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:34:59.264853 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.265168 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.265179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.265189 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.265334 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.265669 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":106,"threads_started":1}
I20260929 18:34:59.268240 18146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46755
I20260929 18:34:59.268487 18146 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:59.268986 18146 heartbeater.cc:511] Master 127.0.98.62:46755 requested a full tablet report, sending...
I20260929 18:34:59.270594 17958 ts_manager.cc:194] Registered new tserver with Master: 248b23fe9efd4fea805210b01ba81106 (127.0.98.1:46499)
I20260929 18:34:59.272185 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51259
W20260929 18:34:59.371456 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.371704 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.371742 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.375908 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.375967 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.376022 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:34:59.380672 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:59.381647 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.382930 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.392025 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.392114 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.392017 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.392349 18149 server_base.cc:1034] running on GCE node
I20260929 18:34:59.392767 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.393356 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.394506 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790706899394496 us; error 32 us; skew 500 ppm
I20260929 18:34:59.396323 18149 webserver.cc:460] Webserver started at http://127.0.98.2:41321/ using document root <none> and password file <none>
I20260929 18:34:59.396876 18149 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.396937 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.397095 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.399075 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e45b05db17e4e668975bf9449778855"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "225ff1aaeb0dddb556756a3524c77ba6"
server_key_iv: "7e550a2843a1025f79aa9b0de0ee6491"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.399580 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e45b05db17e4e668975bf9449778855"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "225ff1aaeb0dddb556756a3524c77ba6"
server_key_iv: "7e550a2843a1025f79aa9b0de0ee6491"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.403115 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:34:59.405660 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.406927 18149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:34:59.407051 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "4e45b05db17e4e668975bf9449778855"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "225ff1aaeb0dddb556756a3524c77ba6"
server_key_iv: "7e550a2843a1025f79aa9b0de0ee6491"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.407203 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:59.426108 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:59.426811 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:59.427008 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.427625 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:59.428689 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:59.428735 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.428773 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:59.428797 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.437683 18149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42985
I20260929 18:34:59.437727 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42985 every 8 connection(s)
I20260929 18:34:59.438572 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:59.445740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18149
I20260929 18:34:59.445878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:34:59.446170 392 external_mini_cluster.cc:1428] Setting key 0875db80c127f79f7c5f401f0eed518c
I20260929 18:34:59.447559 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.440349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38981 (local address 127.0.98.62:46755)
0929 18:34:59.440478 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.440481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.441545 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 18:34:59.442923 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.442927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.443003 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.443089 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.443934 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.444542 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.445645 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.445881 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.445989 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.447108 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.447111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.447113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:34:59.447124 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.447144 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:34:59.447152 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.447205 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.447381 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.447439 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:34:59.448138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--builtin_ntp_servers=127.0.98.20:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:34:59.448606 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.440723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46755 (local address 127.0.98.2:38981)
0929 18:34:59.441357 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.441396 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.442712 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.443258 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.443265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.443345 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.443801 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.443813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.444661 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.444664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.445526 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.445535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.446025 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.446029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.446188 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.446814 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:34:59.446816 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.446882 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:34:59.446958 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.447228 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.447236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.447242 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.447374 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.447718 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260929 18:34:59.449646 18277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46755
I20260929 18:34:59.449855 18277 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:59.450328 18277 heartbeater.cc:511] Master 127.0.98.62:46755 requested a full tablet report, sending...
I20260929 18:34:59.451284 17958 ts_manager.cc:194] Registered new tserver with Master: 4e45b05db17e4e668975bf9449778855 (127.0.98.2:42985)
I20260929 18:34:59.451844 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38981
W20260929 18:34:59.555732 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.555994 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.556031 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.559979 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.560035 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.560088 18280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:34:59.564872 18280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:34:59.565871 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.567298 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.576877 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.576927 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.576876 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.577132 18280 server_base.cc:1034] running on GCE node
I20260929 18:34:59.577590 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.578220 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.579376 18280 hybrid_clock.cc:648] HybridClock initialized: now 1790706899579360 us; error 28 us; skew 500 ppm
I20260929 18:34:59.581187 18280 webserver.cc:460] Webserver started at http://127.0.98.3:38679/ using document root <none> and password file <none>
I20260929 18:34:59.581780 18280 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.581853 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.582113 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.584268 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "dde1301740234b1b925474e1e8275b29"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "e9d99a403d4abcca6ebf731b92cbb374"
server_key_iv: "4ae4c9529350e51e3537a12b0b4357c6"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.584832 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dde1301740234b1b925474e1e8275b29"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "e9d99a403d4abcca6ebf731b92cbb374"
server_key_iv: "4ae4c9529350e51e3537a12b0b4357c6"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.588532 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:59.591032 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.592541 18280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:59.592681 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "dde1301740234b1b925474e1e8275b29"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-fmv8"
server_key: "e9d99a403d4abcca6ebf731b92cbb374"
server_key_iv: "4ae4c9529350e51e3537a12b0b4357c6"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.592794 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:59.611619 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:59.612367 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:59.612560 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.613147 18280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:59.614078 18280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:59.614147 18280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.614213 18280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:59.614257 18280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.623157 18280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37889
I20260929 18:34:59.623220 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37889 every 8 connection(s)
I20260929 18:34:59.624105 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:59.633724 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.626141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57919 (local address 127.0.98.62:46755)
0929 18:34:59.626307 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.626310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.627449 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 18:34:59.628975 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.628978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.629037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.629115 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.630118 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.630651 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.631793 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.632027 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.632136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.633191 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.633195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.633198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:34:59.633211 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.633238 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:34:59.633246 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.633319 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.633520 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.633579 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:34:59.634224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18280
I20260929 18:34:59.634351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:34:59.634639 392 external_mini_cluster.cc:1428] Setting key c3f3b06a176096e044955931b8e1995e
I20260929 18:34:59.634661 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.626545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46755 (local address 127.0.98.3:57919)
0929 18:34:59.627235 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.627294 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.628767 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.629281 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.629287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.629379 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.629977 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.629992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.630775 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.630778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.631669 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.631677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.632216 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.632222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.632378 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.632940 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:34:59.632942 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.632980 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:34:59.633040 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.633332 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.633341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.633394 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.633518 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.633797 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260929 18:34:59.635859 18408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46755
I20260929 18:34:59.636087 18408 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:59.636674 18408 heartbeater.cc:511] Master 127.0.98.62:46755 requested a full tablet report, sending...
I20260929 18:34:59.637797 17957 ts_manager.cc:194] Registered new tserver with Master: dde1301740234b1b925474e1e8275b29 (127.0.98.3:37889)
I20260929 18:34:59.638367 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57919
I20260929 18:34:59.648701 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:59.654403 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.650716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.98.62:46755)
0929 18:34:59.650860 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.650864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.651082 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:34:59.651146 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.651148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.651189 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.651268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.651552 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.652080 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.652975 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.653183 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.653263 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.654006 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.654009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.654011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:34:59.654022 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.654039 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:34:59.654046 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.654102 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.654190 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.654237 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:34:59.657311 17957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:34:59.660362 17957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:34:59.674620 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.668697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37889 (local address 127.0.0.1:59412)
0929 18:34:59.669624 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.669674 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.669829 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.670585 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.670588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.670622 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:59.670884 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.670892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.671870 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.671877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.673171 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.673183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.674061 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.674065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.674169 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.674210 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.674324 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.674440 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":94,"spinlock_wait_cycles":2560,"thread_start_us":105,"threads_started":1}
I20260929 18:34:59.674639 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.668915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.98.3:37889)
0929 18:34:59.669410 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.669429 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.669761 (+ 332us) server_negotiation.cc:407] Connection header received
0929 18:34:59.670006 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.670013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.670239 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.670375 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:59.670996 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.671755 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.673325 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.673939 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.674043 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.674235 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.674366 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.674474 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":151,"threads_started":1}
I20260929 18:34:59.675679 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.668705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.0.98.2:42985)
0929 18:34:59.669543 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.669555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.669964 (+ 409us) server_negotiation.cc:407] Connection header received
0929 18:34:59.670076 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.670082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.670297 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.670444 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:59.671050 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.672281 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.673907 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.675080 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.675197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.675378 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.675425 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.675508 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":114,"threads_started":1}
I20260929 18:34:59.675868 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.669390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46499 (local address 127.0.0.1:38064)
0929 18:34:59.670214 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.670228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.670305 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.670734 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.670737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.670749 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:59.670907 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.670915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.672416 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.672419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.673782 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.673789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.675112 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.675116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.675200 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.675222 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.675291 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.675735 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":201,"thread_start_us":121,"threads_started":1}
I20260929 18:34:59.676354 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.668620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.0.1:40964)
0929 18:34:59.669627 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.669678 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.669813 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.670583 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.670587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.670620 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:59.670914 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.670920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.672416 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.672419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.673757 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.673768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.675216 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.675220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.675307 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.675330 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.675397 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.676223 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":198,"spinlock_wait_cycles":11264,"thread_start_us":77,"threads_started":1}
I20260929 18:34:59.676424 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.669850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.0.98.1:46499)
0929 18:34:59.670304 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.670318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.670362 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:34:59.670431 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.670436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.670601 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.670741 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:59.671047 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.672281 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.673905 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.674723 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.674832 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.675049 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.676172 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.676273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":100,"threads_started":1}
I20260929 18:34:59.677942 18342 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 (DEFAULT_TABLE table=test-table [id=ca447bfb39c7480cbc02642c7d39757d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:59.679080 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce436043e8046f7977b484b58ebb9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.678966 18211 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 (DEFAULT_TABLE table=test-table [id=ca447bfb39c7480cbc02642c7d39757d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:59.679961 18211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce436043e8046f7977b484b58ebb9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.680778 18080 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 (DEFAULT_TABLE table=test-table [id=ca447bfb39c7480cbc02642c7d39757d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:59.681540 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce436043e8046f7977b484b58ebb9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.686753 18425 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Bootstrap starting.
I20260929 18:34:59.687914 18426 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Bootstrap starting.
I20260929 18:34:59.688468 18427 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Bootstrap starting.
I20260929 18:34:59.689189 18425 tablet_bootstrap.cc:654] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:59.690026 18426 tablet_bootstrap.cc:654] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:59.690040 18425 log.cc:824] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:59.690601 18427 tablet_bootstrap.cc:654] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:59.690804 18426 log.cc:824] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:59.691411 18427 log.cc:824] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:59.692408 18425 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: No bootstrap required, opened a new log
I20260929 18:34:59.692606 18425 ts_tablet_manager.cc:1397] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:34:59.692744 18426 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: No bootstrap required, opened a new log
I20260929 18:34:59.692924 18426 ts_tablet_manager.cc:1397] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:34:59.693346 18427 tablet_bootstrap.cc:492] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: No bootstrap required, opened a new log
I20260929 18:34:59.693517 18427 ts_tablet_manager.cc:1397] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:34:59.695837 18425 raft_consensus.cc:348] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.696138 18425 raft_consensus.cc:374] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:59.696223 18425 raft_consensus.cc:729] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde1301740234b1b925474e1e8275b29, State: Initialized, Role: FOLLOWER
I20260929 18:34:59.696722 18425 consensus_queue.cc:260] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.697073 18426 raft_consensus.cc:348] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.697300 18426 raft_consensus.cc:374] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:59.697347 18426 raft_consensus.cc:729] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e45b05db17e4e668975bf9449778855, State: Initialized, Role: FOLLOWER
I20260929 18:34:59.697765 18425 ts_tablet_manager.cc:1428] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 18:34:59.697773 18426 consensus_queue.cc:260] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.697749 18427 raft_consensus.cc:348] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.697968 18427 raft_consensus.cc:374] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:59.698033 18427 raft_consensus.cc:729] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248b23fe9efd4fea805210b01ba81106, State: Initialized, Role: FOLLOWER
I20260929 18:34:59.698130 18408 heartbeater.cc:503] Master 127.0.98.62:46755 was elected leader, sending a full tablet report...
I20260929 18:34:59.698419 18427 consensus_queue.cc:260] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:34:59.699110 18146 heartbeater.cc:503] Master 127.0.98.62:46755 was elected leader, sending a full tablet report...
I20260929 18:34:59.699316 18427 ts_tablet_manager.cc:1428] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:34:59.700245 18277 heartbeater.cc:503] Master 127.0.98.62:46755 was elected leader, sending a full tablet report...
I20260929 18:34:59.700563 18426 ts_tablet_manager.cc:1428] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260929 18:34:59.756021 18147 tablet.cc:2369] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:34:59.875273 18409 tablet.cc:2369] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:34:59.939670 18278 tablet.cc:2369] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:00.015568 18431 raft_consensus.cc:482] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:00.015782 18431 raft_consensus.cc:504] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:35:00.017118 18431 leader_election.cc:288] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e45b05db17e4e668975bf9449778855 (127.0.98.2:42985), 248b23fe9efd4fea805210b01ba81106 (127.0.98.1:46499)
I20260929 18:35:00.020665 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.017399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.98.3:52101)
0929 18:35:00.017530 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.017548 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.017663 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.017957 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.017960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.017978 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:00.018098 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.018105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.018828 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.018833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.019607 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.019615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.020260 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.020263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.020367 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.020384 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.020468 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.020537 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 18:35:00.020665 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.017419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52101 (local address 127.0.98.2:42985)
0929 18:35:00.017551 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.017555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.017614 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:35:00.017787 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.017791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.017842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.017943 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:00.018215 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.018717 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.019736 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.020121 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.020257 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.020318 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.020483 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.020547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:35:00.021189 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.017875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57795 (local address 127.0.98.1:46499)
0929 18:35:00.018004 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.018008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.018166 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:35:00.018289 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.018296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.018340 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.018421 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:00.018681 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.019184 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.020255 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.020640 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.020757 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.020811 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.021000 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.021075 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:00.021350 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce436043e8046f7977b484b58ebb9f7" candidate_uuid: "dde1301740234b1b925474e1e8275b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e45b05db17e4e668975bf9449778855" is_pre_election: true
I20260929 18:35:00.021548 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.017704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46499 (local address 127.0.98.3:57795)
0929 18:35:00.018086 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.018097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.018200 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.018433 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.018436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.018451 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:00.018576 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.018581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.019297 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.019300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.020107 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.020116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.020764 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.020767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.020862 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.020886 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.021329 (+ 443us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.021415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260929 18:35:00.021680 18231 raft_consensus.cc:2454] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dde1301740234b1b925474e1e8275b29 in term 0.
I20260929 18:35:00.022132 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce436043e8046f7977b484b58ebb9f7" candidate_uuid: "dde1301740234b1b925474e1e8275b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248b23fe9efd4fea805210b01ba81106" is_pre_election: true
I20260929 18:35:00.022152 18296 leader_election.cc:302] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e45b05db17e4e668975bf9449778855, dde1301740234b1b925474e1e8275b29; no voters:
I20260929 18:35:00.022416 18100 raft_consensus.cc:2454] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dde1301740234b1b925474e1e8275b29 in term 0.
I20260929 18:35:00.022516 18431 raft_consensus.cc:2781] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:00.022629 18431 raft_consensus.cc:482] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:00.022686 18431 raft_consensus.cc:3037] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.023723 18431 raft_consensus.cc:504] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:35:00.024053 18431 leader_election.cc:288] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [CANDIDATE]: Term 1 election: Requested vote from peers 4e45b05db17e4e668975bf9449778855 (127.0.98.2:42985), 248b23fe9efd4fea805210b01ba81106 (127.0.98.1:46499)
I20260929 18:35:00.024395 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce436043e8046f7977b484b58ebb9f7" candidate_uuid: "dde1301740234b1b925474e1e8275b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e45b05db17e4e668975bf9449778855"
I20260929 18:35:00.024499 18231 raft_consensus.cc:3037] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.025431 18231 raft_consensus.cc:2454] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dde1301740234b1b925474e1e8275b29 in term 1.
I20260929 18:35:00.025712 18296 leader_election.cc:302] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e45b05db17e4e668975bf9449778855, dde1301740234b1b925474e1e8275b29; no voters:
I20260929 18:35:00.025835 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce436043e8046f7977b484b58ebb9f7" candidate_uuid: "dde1301740234b1b925474e1e8275b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248b23fe9efd4fea805210b01ba81106"
I20260929 18:35:00.025878 18431 raft_consensus.cc:2781] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:00.025991 18100 raft_consensus.cc:3037] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.026119 18431 raft_consensus.cc:686] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 1 LEADER]: Becoming Leader. State: Replica: dde1301740234b1b925474e1e8275b29, State: Running, Role: LEADER
I20260929 18:35:00.026444 18431 consensus_queue.cc:237] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } }
I20260929 18:35:00.027195 18100 raft_consensus.cc:2454] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dde1301740234b1b925474e1e8275b29 in term 1.
I20260929 18:35:00.029711 17956 catalog_manager.cc:5495] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 reported cstate change: term changed from 0 to 1, leader changed from <none> to dde1301740234b1b925474e1e8275b29 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dde1301740234b1b925474e1e8275b29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dde1301740234b1b925474e1e8275b29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:00.124119 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.120868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.98.3:37889)
0929 18:35:00.121008 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.121011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.121028 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:00.121103 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.121105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.121157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.121246 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:00.121471 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.121978 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.122776 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.122999 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.123077 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.123332 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:00.123749 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:00.123841 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.123902 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.123945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:35:00.130100 18231 raft_consensus.cc:1261] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 1 FOLLOWER]: Refusing update from remote peer dde1301740234b1b925474e1e8275b29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:00.130100 18100 raft_consensus.cc:1261] T 0ce436043e8046f7977b484b58ebb9f7 P 248b23fe9efd4fea805210b01ba81106 [term 1 FOLLOWER]: Refusing update from remote peer dde1301740234b1b925474e1e8275b29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:00.131045 18431 consensus_queue.cc:1035] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e45b05db17e4e668975bf9449778855" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:00.131300 18435 consensus_queue.cc:1035] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248b23fe9efd4fea805210b01ba81106" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:00.140172 18439 mvcc.cc:205] Tried to move back new op lower bound from 7334735462925168640 to 7334735462510796800. Current Snapshot: MvccSnapshot[applied={T|T < 7334735462925168640}]
I20260929 18:35:00.147111 17958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:00.147286 17958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:00.149695 17958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693dac4f1f844e089b4f751836329526: Sending DeleteTablet for 3 replicas of tablet 0ce436043e8046f7977b484b58ebb9f7
I20260929 18:35:00.150480 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:00 UTC) from {username='slave'} at 127.0.0.1:59412
I20260929 18:35:00.150626 18211 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:00 UTC) from {username='slave'} at 127.0.0.1:40964
I20260929 18:35:00.150642 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce436043e8046f7977b484b58ebb9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:00 UTC) from {username='slave'} at 127.0.0.1:38064
I20260929 18:35:00.151084 18448 tablet_replica.cc:330] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: stopping tablet replica
I20260929 18:35:00.151403 18448 raft_consensus.cc:2229] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:00.151721 18449 tablet_replica.cc:330] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: stopping tablet replica
I20260929 18:35:00.151779 18448 raft_consensus.cc:2258] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:00.151826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18018
I20260929 18:35:00.152776 18449 raft_consensus.cc:2229] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:00.153064 18449 raft_consensus.cc:2258] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:00.153256 18448 ts_tablet_manager.cc:1905] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:00.155424 18448 ts_tablet_manager.cc:1918] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:00.155489 18448 log.cc:1196] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/0ce436043e8046f7977b484b58ebb9f7
I20260929 18:35:00.155706 18448 ts_tablet_manager.cc:1939] T 0ce436043e8046f7977b484b58ebb9f7 P dde1301740234b1b925474e1e8275b29: Deleting consensus metadata
I20260929 18:35:00.156874 17944 catalog_manager.cc:4841] TS dde1301740234b1b925474e1e8275b29 (127.0.98.3:37889): tablet 0ce436043e8046f7977b484b58ebb9f7 (table test-table [id=ca447bfb39c7480cbc02642c7d39757d]) successfully deleted
I20260929 18:35:00.158004 18449 ts_tablet_manager.cc:1905] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:00.160310 18449 ts_tablet_manager.cc:1918] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:00.160427 18449 log.cc:1196] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/0ce436043e8046f7977b484b58ebb9f7
I20260929 18:35:00.160701 18449 ts_tablet_manager.cc:1939] T 0ce436043e8046f7977b484b58ebb9f7 P 4e45b05db17e4e668975bf9449778855: Deleting consensus metadata
W20260929 18:35:00.161666 17943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46499 (error 108)
I20260929 18:35:00.162115 17943 catalog_manager.cc:4841] TS 4e45b05db17e4e668975bf9449778855 (127.0.98.2:42985): tablet 0ce436043e8046f7977b484b58ebb9f7 (table test-table [id=ca447bfb39c7480cbc02642c7d39757d]) successfully deleted
I20260929 18:35:00.162638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18149
I20260929 18:35:00.162684 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.162394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46499 (local address 127.0.0.1:38080)
0929 18:35:00.162545 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.162614 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:35:00.164882 17943 catalog_manager.cc:4584] TS 248b23fe9efd4fea805210b01ba81106 (127.0.98.1:46499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ce436043e8046f7977b484b58ebb9f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46499: connect: Connection refused (error 111)
I20260929 18:35:00.169374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18280
I20260929 18:35:00.175043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 17927
2026-09-29T18:35:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1363 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:00 dist-test-slave-fmv8 krb5kdc[18454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:00 dist-test-slave-fmv8 krb5kdc[18454](info): set up 2 sockets
Sep 29 18:35:00 dist-test-slave-fmv8 krb5kdc[18454](info): commencing operation
krb5kdc: starting...
W20260929 18:35:01.216255 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:01.339489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:01.449535 18467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:01.449805 18467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:01.449847 18467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:01.453842 18467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:01.453905 18467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:01.453924 18467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:01.453941 18467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:01.453956 18467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:01.459331 18467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:01.460453 18467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:01.461491 18467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:01.470180 18473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.470120 18475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.470120 18472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:01.470448 18467 server_base.cc:1034] running on GCE node
I20260929 18:35:01.471318 18467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:01.472622 18467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:01.473819 18467 hybrid_clock.cc:648] HybridClock initialized: now 1790706901473786 us; error 56 us; skew 500 ppm
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:01.480502 18467 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:01.481499 18467 webserver.cc:460] Webserver started at http://127.0.98.62:39675/ using document root <none> and password file <none>
I20260929 18:35:01.482102 18467 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:01.482197 18467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:01.482430 18467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:01.484558 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "1c90652220bd491fa0cb644448fb3b7c"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "45207977265ffca33c2405ee827c0dc4"
server_key_iv: "fd3df76b1c67f544ced61aeca8ac84ab"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.485124 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c90652220bd491fa0cb644448fb3b7c"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "45207977265ffca33c2405ee827c0dc4"
server_key_iv: "fd3df76b1c67f544ced61aeca8ac84ab"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.488869 18467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:01.491312 18482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:01.492712 18467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:01.492849 18467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "1c90652220bd491fa0cb644448fb3b7c"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "45207977265ffca33c2405ee827c0dc4"
server_key_iv: "fd3df76b1c67f544ced61aeca8ac84ab"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.492962 18467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:01.507831 18467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:01.511021 18467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:01.511300 18467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:01.518621 18467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46429
I20260929 18:35:01.518685 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46429 every 8 connection(s)
I20260929 18:35:01.519614 18467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:01.522444 18535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:01.526170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18467
I20260929 18:35:01.526315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:01.526643 392 external_mini_cluster.cc:1428] Setting key 6f0a535d0c75d689160e2fc4a85627ee
I20260929 18:35:01.528543 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: Bootstrap starting.
I20260929 18:35:01.531037 18535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:01.531827 18535 log.cc:824] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:01.533964 18535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: No bootstrap required, opened a new log
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:01.538156 18535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } }
I20260929 18:35:01.538388 18535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:01.538451 18535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c90652220bd491fa0cb644448fb3b7c, State: Initialized, Role: FOLLOWER
I20260929 18:35:01.538915 18535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } }
I20260929 18:35:01.539070 18535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:01.539151 18535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:01.539271 18535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:01.540239 18535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } }
I20260929 18:35:01.540632 18535 leader_election.cc:302] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c90652220bd491fa0cb644448fb3b7c; no voters:
I20260929 18:35:01.541042 18535 leader_election.cc:288] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:01.541118 18540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:01.541350 18540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [term 1 LEADER]: Becoming Leader. State: Replica: 1c90652220bd491fa0cb644448fb3b7c, State: Running, Role: LEADER
I20260929 18:35:01.541726 18540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } }
I20260929 18:35:01.541774 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:01.528030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.0.98.62:46429)
0929 18:35:01.528513 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:01.528529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:01.528566 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:35:01.529242 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:01.529258 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:01.529578 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:01.529863 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:01.530745 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.531829 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.532964 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.533451 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.533587 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:01.536569 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:01.536574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:01.536587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:01.536632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:01.538480 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:01.539030 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:01.539036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:01.539046 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:01.539218 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:01.539506 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:01.539511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:01.539512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:01.539923 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:01.540142 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:01.540418 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:35:01.540735 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":207,"threads_started":1}
I20260929 18:35:01.542448 18535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:01.545378 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c90652220bd491fa0cb644448fb3b7c. Latest consensus state: current_term: 1 leader_uuid: "1c90652220bd491fa0cb644448fb3b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } } }
I20260929 18:35:01.545518 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:01.545850 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c90652220bd491fa0cb644448fb3b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c90652220bd491fa0cb644448fb3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46429 } } }
I20260929 18:35:01.545941 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:01.547775 18554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:01.547858 18554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:01.547932 18556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:01.548820 18556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:01.554493 18556 catalog_manager.cc:1347] Generated new cluster ID: f09c7ec2e32143fcbb6a8a4a4988af55
I20260929 18:35:01.554569 18556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:01.580863 18556 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:01.581914 18556 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:01.586059 18556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: Generated new TSK 0
I20260929 18:35:01.586761 18556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:35:01.619249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:01.735075 18561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:01.735345 18561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:01.735390 18561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:01.739436 18561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:01.739499 18561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:01.739558 18561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:35:01.745205 18561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:01.746266 18561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:01.747296 18561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:01.756989 18566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.756999 18567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.756954 18569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:01.757055 18561 server_base.cc:1034] running on GCE node
I20260929 18:35:01.757583 18561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:01.758214 18561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:01.759371 18561 hybrid_clock.cc:648] HybridClock initialized: now 1790706901759343 us; error 44 us; skew 500 ppm
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:01.762986 18561 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:35:01.763994 18561 webserver.cc:460] Webserver started at http://127.0.98.1:37905/ using document root <none> and password file <none>
I20260929 18:35:01.764605 18561 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:01.764701 18561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:01.764905 18561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:01.766971 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "8deda9dbef4344edb8303f8977d75e73"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "174b777d6eaf6d4f20e96e09f151560a"
server_key_iv: "0fde969f08ed5a04824c11789bd6be2d"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.767537 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8deda9dbef4344edb8303f8977d75e73"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "174b777d6eaf6d4f20e96e09f151560a"
server_key_iv: "0fde969f08ed5a04824c11789bd6be2d"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.771652 18561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:01.774184 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:01.775508 18561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:35:01.775700 18561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "8deda9dbef4344edb8303f8977d75e73"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "174b777d6eaf6d4f20e96e09f151560a"
server_key_iv: "0fde969f08ed5a04824c11789bd6be2d"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.775859 18561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:01.803912 18561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:01.807215 18561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:01.807499 18561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:01.808130 18561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:01.809005 18561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:01.809074 18561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:01.809141 18561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:01.809190 18561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:01.818135 18561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33301
I20260929 18:35:01.818197 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33301 every 8 connection(s)
I20260929 18:35:01.819084 18561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:01.826717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18561
I20260929 18:35:01.826858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:01.827165 392 external_mini_cluster.cc:1428] Setting key 3d615d57448547650ac34423db7b7c20
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:01.833565 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:01.821164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44539 (local address 127.0.98.62:46429)
0929 18:35:01.821296 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:01.821300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:01.822143 (+ 843us) server_negotiation.cc:407] Connection header received
0929 18:35:01.823475 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:01.823478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:01.823537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:01.823618 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:01.824919 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.825475 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.826769 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.827126 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.827243 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:01.829909 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:01.829912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:01.829914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:01.829938 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:01.831406 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:01.831936 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:01.831940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:01.831942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:01.832023 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:01.832432 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:01.832436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:01.832438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:01.832628 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:01.832742 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:01.833264 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 18:35:01.833400 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:01.834628 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:01.821468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46429 (local address 127.0.98.1:44539)
0929 18:35:01.821921 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:01.821970 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:01.823257 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:01.823818 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:01.823825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:01.824266 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:01.824767 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:01.824781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.825616 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.825619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:01.826602 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:01.826614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:01.827296 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:01.827300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:01.827495 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:01.828182 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:01.828210 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:01.829736 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:01.831532 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:01.831538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:01.831555 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:01.831830 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:01.832114 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:01.832118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:01.832120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:01.832326 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:01.832733 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:01.832740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:01.833016 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 18:35:01.833205 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:35:01.833569 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260929 18:35:01.835989 18690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46429
I20260929 18:35:01.836315 18690 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:35:01.836946 18690 heartbeater.cc:511] Master 127.0.98.62:46429 requested a full tablet report, sending...
I20260929 18:35:01.838402 18499 ts_manager.cc:194] Registered new tserver with Master: 8deda9dbef4344edb8303f8977d75e73 (127.0.98.1:33301)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:35:01.839571 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:35:01.855124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:01.967347 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:01.967659 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:01.967707 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:01.971987 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:01.972064 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:01.972126 18695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:35:01.977624 18695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:01.978796 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:01.979890 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:01.989744 18700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.989773 18701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:01.989744 18703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:01.990020 18695 server_base.cc:1034] running on GCE node
I20260929 18:35:01.990489 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:01.991202 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:01.992398 18695 hybrid_clock.cc:648] HybridClock initialized: now 1790706901992376 us; error 38 us; skew 500 ppm
Sep 29 18:35:01 dist-test-slave-fmv8 krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:01.995247 18695 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:35:01.996245 18695 webserver.cc:460] Webserver started at http://127.0.98.2:37643/ using document root <none> and password file <none>
I20260929 18:35:01.996861 18695 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:01.996940 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:01.997115 18695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:01.999233 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "016769fe7fa046d9814241fcc84b8022"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "ae8cbe9a83ed098ed65c47b5c7a1fa23"
server_key_iv: "742d5916e73dc074c8fa81c75ab528b6"
server_key_version: "encryptionkey@0"
I20260929 18:35:01.999751 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "016769fe7fa046d9814241fcc84b8022"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "ae8cbe9a83ed098ed65c47b5c7a1fa23"
server_key_iv: "742d5916e73dc074c8fa81c75ab528b6"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.003582 18695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:02.006129 18710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.007618 18695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:35:02.007767 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "016769fe7fa046d9814241fcc84b8022"
format_stamp: "Formatted at 2026-09-29 18:35:01 on dist-test-slave-fmv8"
server_key: "ae8cbe9a83ed098ed65c47b5c7a1fa23"
server_key_iv: "742d5916e73dc074c8fa81c75ab528b6"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.007880 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:02.032964 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:02.038681 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:02.039003 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:02.039656 18695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:02.040563 18695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:02.040611 18695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.040693 18695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:02.040746 18695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.050261 18695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43819
I20260929 18:35:02.050280 18823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43819 every 8 connection(s)
I20260929 18:35:02.051201 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:02.051903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18695
I20260929 18:35:02.052044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:02.052408 392 external_mini_cluster.cc:1428] Setting key 84a694b0a9c723a4fc766d9fed8bd009
Sep 29 18:35:02 dist-test-slave-fmv8 krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 18:35:02.066056 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.053203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53189 (local address 127.0.98.62:46429)
0929 18:35:02.053347 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.053351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.054595 (+ 1244us) server_negotiation.cc:407] Connection header received
0929 18:35:02.055492 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.055495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.055581 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.055681 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:02.056899 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.057428 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.058565 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.058791 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.058898 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.061523 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:02.061527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:02.061530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:02.061563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:02.063014 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.063692 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.063695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.063697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.063801 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.064335 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.064342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.064344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.064553 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:02.064644 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.065412 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.065730 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:35:02.066730 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.053663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46429 (local address 127.0.98.2:53189)
0929 18:35:02.054459 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.054500 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.055295 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.055862 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.055869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.056292 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:02.056768 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.056781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.057551 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.057555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.058441 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.058450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.058935 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.058940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.059135 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.059779 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:02.059795 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:02.061319 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:02.063220 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:02.063229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:02.063247 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:02.063556 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:02.063969 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:02.063971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:02.063973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:02.064234 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:02.064683 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:02.064691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:02.065044 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.065272 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.065725 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":121,"threads_started":1}
I20260929 18:35:02.068094 18824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46429
I20260929 18:35:02.068395 18824 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:02.068922 18824 heartbeater.cc:511] Master 127.0.98.62:46429 requested a full tablet report, sending...
I20260929 18:35:02.070070 18499 ts_manager.cc:194] Registered new tserver with Master: 016769fe7fa046d9814241fcc84b8022 (127.0.98.2:43819)
I20260929 18:35:02.070952 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:35:02.080916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:02.189785 18829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:02.190024 18829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:02.190095 18829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:02.194165 18829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:02.194221 18829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:02.194312 18829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:35:02.199725 18829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:02.200743 18829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:02.201719 18829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:02.211146 18835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.211160 18837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.211160 18834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:02.211485 18829 server_base.cc:1034] running on GCE node
I20260929 18:35:02.211973 18829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:02.212693 18829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:02.213871 18829 hybrid_clock.cc:648] HybridClock initialized: now 1790706902213832 us; error 54 us; skew 500 ppm
Sep 29 18:35:02 dist-test-slave-fmv8 krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:02.216822 18829 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:35:02.217809 18829 webserver.cc:460] Webserver started at http://127.0.98.3:36395/ using document root <none> and password file <none>
I20260929 18:35:02.218376 18829 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:02.218461 18829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:02.218695 18829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:02.220790 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "a078711758aa4a6199c539ecd4301bd5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "2dfae3b2fe349dd9a2783c2ba6c5cbd6"
server_key_iv: "71f94d69066aabb8f87ffa60b9282487"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.221356 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a078711758aa4a6199c539ecd4301bd5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "2dfae3b2fe349dd9a2783c2ba6c5cbd6"
server_key_iv: "71f94d69066aabb8f87ffa60b9282487"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.224959 18829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:02.227372 18844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.228674 18829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:02.228807 18829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "a078711758aa4a6199c539ecd4301bd5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "2dfae3b2fe349dd9a2783c2ba6c5cbd6"
server_key_iv: "71f94d69066aabb8f87ffa60b9282487"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.229120 18829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:02.269486 18829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:02.272586 18829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:02.272784 18829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:02.273373 18829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:02.274195 18829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:02.274262 18829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.274328 18829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:02.274374 18829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.283418 18829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38245
I20260929 18:35:02.283437 18957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38245 every 8 connection(s)
I20260929 18:35:02.284364 18829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:02.287413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 18829
I20260929 18:35:02.287590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:02.287972 392 external_mini_cluster.cc:1428] Setting key 07d0c998d41eb7f3885216018cefe1fc
Sep 29 18:35:02 dist-test-slave-fmv8 krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:02.299330 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.286116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44341 (local address 127.0.98.62:46429)
0929 18:35:02.286260 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.286264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.287131 (+ 867us) server_negotiation.cc:407] Connection header received
0929 18:35:02.288821 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.288824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.288896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.288991 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:02.290545 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.291081 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.292475 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.292702 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.292797 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.295377 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:02.295414 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:02.295421 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:02.295457 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:02.297289 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.297812 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.297815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.297816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.297893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.298326 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.298329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.298330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.298515 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:02.298599 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.299082 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.299209 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:35:02.300263 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.286457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46429 (local address 127.0.98.3:44341)
0929 18:35:02.286938 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.286982 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.288575 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.289284 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.289292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.289767 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:02.290410 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.290434 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.291237 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.291240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.292315 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.292325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.292849 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.292852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.292997 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.293621 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:02.293643 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:02.295193 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:02.297444 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:02.297449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:02.297461 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:02.297723 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:02.297999 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:02.298002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:02.298004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:02.298207 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:02.298631 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:02.298636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:02.298874 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.299043 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.299311 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260929 18:35:02.301498 18958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46429
I20260929 18:35:02.301792 18958 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:02.302369 18958 heartbeater.cc:511] Master 127.0.98.62:46429 requested a full tablet report, sending...
I20260929 18:35:02.303444 18498 ts_manager.cc:194] Registered new tserver with Master: a078711758aa4a6199c539ecd4301bd5 (127.0.98.3:38245)
I20260929 18:35:02.304068 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44341
I20260929 18:35:02.313978 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:02.324206 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.315925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.0.98.62:46429)
0929 18:35:02.316083 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.316089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.316319 (+ 230us) server_negotiation.cc:407] Connection header received
0929 18:35:02.316463 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.316466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.316512 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.316617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:02.317324 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.317844 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.318786 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.319026 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.319135 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.320835 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:02.320839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:02.320842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:02.320870 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:02.322269 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.322825 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.322828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.322830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.322914 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.323235 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.323238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.323240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.323462 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:02.323621 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.323884 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.323981 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:35:02.327574 18498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:02.330534 18498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:02.345866 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.339373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.3:38245)
0929 18:35:02.340136 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.340178 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.340205 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:35:02.340319 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.340326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.340555 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.341100 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:02.341271 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.342461 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.344404 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.345131 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.345237 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.345368 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.345593 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.345696 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":127,"threads_started":1}
I20260929 18:35:02.345857 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.339308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33301 (local address 127.0.0.1:50200)
0929 18:35:02.340548 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.340564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.340664 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.341224 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.341227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.341244 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:02.341362 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.341369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.342644 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.342647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.344120 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.344129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.345432 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.345436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.345548 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.345573 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.345655 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.345719 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":105,"thread_start_us":132,"threads_started":1}
I20260929 18:35:02.345865 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.339261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38245 (local address 127.0.0.1:48472)
0929 18:35:02.339704 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.339751 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.339854 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.340747 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.340754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.340780 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:02.340936 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.340944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.342580 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.342587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.344263 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.344274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.345256 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.345260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.345387 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.345430 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.345549 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.345680 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":224,"thread_start_us":77,"threads_started":1}
I20260929 18:35:02.346283 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.340174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.98.1:33301)
0929 18:35:02.340449 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.340455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.340771 (+ 316us) server_negotiation.cc:407] Connection header received
0929 18:35:02.340861 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.340869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.341063 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.341241 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:02.341483 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.342543 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.344265 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.345105 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.345221 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.345368 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.346056 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.346159 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260929 18:35:02.346867 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.340037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43819 (local address 127.0.0.1:36770)
0929 18:35:02.340945 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.340958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.341031 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.341576 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.341579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.341598 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:02.341690 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.341697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.343820 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.343825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.344893 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.344901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.346258 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.346416 (+ 158us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.346525 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.346576 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.346669 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.346742 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":7,"thread_start_us":111,"threads_started":1}
I20260929 18:35:02.347381 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.340602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.98.2:43819)
0929 18:35:02.341171 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.341182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.341203 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:35:02.341272 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.341277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.341435 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.341579 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:02.341785 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.342918 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.345014 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.345909 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.346027 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.346184 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.347110 (+ 926us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.347216 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":112,"threads_started":1}
I20260929 18:35:02.349378 18892 tablet_service.cc:1467] Processing CreateTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 (DEFAULT_TABLE table=test-table [id=a2609b84d83b4bc79bb4ba8521d57a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:02.349375 18624 tablet_service.cc:1467] Processing CreateTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 (DEFAULT_TABLE table=test-table [id=a2609b84d83b4bc79bb4ba8521d57a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:02.349866 18758 tablet_service.cc:1467] Processing CreateTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 (DEFAULT_TABLE table=test-table [id=a2609b84d83b4bc79bb4ba8521d57a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:02.350497 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316cd650c98845f7ae3bf8936cc4e012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:02.350497 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316cd650c98845f7ae3bf8936cc4e012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:02.351007 18758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316cd650c98845f7ae3bf8936cc4e012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:02.357326 18975 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: Bootstrap starting.
I20260929 18:35:02.358609 18976 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: Bootstrap starting.
I20260929 18:35:02.359965 18975 tablet_bootstrap.cc:654] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:02.360373 18976 tablet_bootstrap.cc:654] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:02.360473 18977 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Bootstrap starting.
I20260929 18:35:02.360989 18975 log.cc:824] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:02.361209 18976 log.cc:824] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:02.362663 18977 tablet_bootstrap.cc:654] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:02.363114 18975 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: No bootstrap required, opened a new log
I20260929 18:35:02.363314 18975 ts_tablet_manager.cc:1397] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:35:02.363428 18976 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: No bootstrap required, opened a new log
I20260929 18:35:02.363597 18976 ts_tablet_manager.cc:1397] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:35:02.363860 18977 log.cc:824] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:02.365631 18977 tablet_bootstrap.cc:492] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: No bootstrap required, opened a new log
I20260929 18:35:02.365828 18977 ts_tablet_manager.cc:1397] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:35:02.367795 18976 raft_consensus.cc:348] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.368047 18976 raft_consensus.cc:374] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:02.368134 18976 raft_consensus.cc:729] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8deda9dbef4344edb8303f8977d75e73, State: Initialized, Role: FOLLOWER
I20260929 18:35:02.367415 18975 raft_consensus.cc:348] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.368310 18975 raft_consensus.cc:374] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:02.368427 18975 raft_consensus.cc:729] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016769fe7fa046d9814241fcc84b8022, State: Initialized, Role: FOLLOWER
I20260929 18:35:02.368752 18976 consensus_queue.cc:260] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.368798 18977 raft_consensus.cc:348] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.368913 18975 consensus_queue.cc:260] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.369063 18977 raft_consensus.cc:374] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:02.369156 18977 raft_consensus.cc:729] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a078711758aa4a6199c539ecd4301bd5, State: Initialized, Role: FOLLOWER
I20260929 18:35:02.369575 18824 heartbeater.cc:503] Master 127.0.98.62:46429 was elected leader, sending a full tablet report...
I20260929 18:35:02.369664 18977 consensus_queue.cc:260] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.369797 18976 ts_tablet_manager.cc:1428] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260929 18:35:02.370087 18690 heartbeater.cc:503] Master 127.0.98.62:46429 was elected leader, sending a full tablet report...
I20260929 18:35:02.370245 18975 ts_tablet_manager.cc:1428] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:35:02.370649 18977 ts_tablet_manager.cc:1428] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 18:35:02.371253 18958 heartbeater.cc:503] Master 127.0.98.62:46429 was elected leader, sending a full tablet report...
I20260929 18:35:02.441547 18982 raft_consensus.cc:482] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:02.441979 18982 raft_consensus.cc:504] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.443753 18982 leader_election.cc:288] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a078711758aa4a6199c539ecd4301bd5 (127.0.98.3:38245), 8deda9dbef4344edb8303f8977d75e73 (127.0.98.1:33301)
I20260929 18:35:02.448915 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.444252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50335 (local address 127.0.98.1:33301)
0929 18:35:02.444395 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.444402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.444569 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:35:02.444677 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.444680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.444732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.444799 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:02.445082 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.445794 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.447569 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.448263 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.448352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.448404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.448705 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.448791 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:35:02.448915 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.444106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34287 (local address 127.0.98.3:38245)
0929 18:35:02.444246 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.444253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.444269 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:35:02.444469 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.444472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.444517 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.444627 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:02.444929 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.445713 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.447579 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.448213 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.448338 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.448404 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.448705 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.448784 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:35:02.448922 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.443966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38245 (local address 127.0.98.2:34287)
0929 18:35:02.444197 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.444213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.444328 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.444658 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.444664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.444681 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:02.444813 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.444821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.445878 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.445882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.447429 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.447440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.448360 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.448364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.448537 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.448573 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.448654 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.448759 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":198}
I20260929 18:35:02.448931 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.444061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33301 (local address 127.0.98.2:50335)
0929 18:35:02.444483 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.444493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:02.444572 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:02.444835 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:02.444838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:02.444857 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:02.444969 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.444980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.445899 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.445902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.447433 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:02.447440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.448367 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.448370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:02.448493 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.448567 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 18:35:02.448652 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:35:02.448759 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":176,"thread_start_us":161,"threads_started":1}
I20260929 18:35:02.449574 18644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316cd650c98845f7ae3bf8936cc4e012" candidate_uuid: "016769fe7fa046d9814241fcc84b8022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8deda9dbef4344edb8303f8977d75e73" is_pre_election: true
I20260929 18:35:02.449908 18644 raft_consensus.cc:2454] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016769fe7fa046d9814241fcc84b8022 in term 0.
I20260929 18:35:02.449906 18912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316cd650c98845f7ae3bf8936cc4e012" candidate_uuid: "016769fe7fa046d9814241fcc84b8022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a078711758aa4a6199c539ecd4301bd5" is_pre_election: true
I20260929 18:35:02.450169 18912 raft_consensus.cc:2454] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016769fe7fa046d9814241fcc84b8022 in term 0.
I20260929 18:35:02.450475 18712 leader_election.cc:302] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016769fe7fa046d9814241fcc84b8022, 8deda9dbef4344edb8303f8977d75e73; no voters:
I20260929 18:35:02.450778 18982 raft_consensus.cc:2781] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:02.450855 18982 raft_consensus.cc:482] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:02.450886 18982 raft_consensus.cc:3037] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:02.451929 18982 raft_consensus.cc:504] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.452318 18982 leader_election.cc:288] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [CANDIDATE]: Term 1 election: Requested vote from peers a078711758aa4a6199c539ecd4301bd5 (127.0.98.3:38245), 8deda9dbef4344edb8303f8977d75e73 (127.0.98.1:33301)
I20260929 18:35:02.452678 18912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316cd650c98845f7ae3bf8936cc4e012" candidate_uuid: "016769fe7fa046d9814241fcc84b8022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a078711758aa4a6199c539ecd4301bd5"
I20260929 18:35:02.452703 18644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316cd650c98845f7ae3bf8936cc4e012" candidate_uuid: "016769fe7fa046d9814241fcc84b8022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8deda9dbef4344edb8303f8977d75e73"
I20260929 18:35:02.452777 18912 raft_consensus.cc:3037] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:02.452847 18644 raft_consensus.cc:3037] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:02.453838 18644 raft_consensus.cc:2454] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016769fe7fa046d9814241fcc84b8022 in term 1.
I20260929 18:35:02.453840 18912 raft_consensus.cc:2454] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016769fe7fa046d9814241fcc84b8022 in term 1.
I20260929 18:35:02.454151 18711 leader_election.cc:302] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016769fe7fa046d9814241fcc84b8022, a078711758aa4a6199c539ecd4301bd5; no voters:
I20260929 18:35:02.454411 18982 raft_consensus.cc:2781] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:02.454653 18982 raft_consensus.cc:686] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 1 LEADER]: Becoming Leader. State: Replica: 016769fe7fa046d9814241fcc84b8022, State: Running, Role: LEADER
I20260929 18:35:02.454993 18982 consensus_queue.cc:237] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } }
I20260929 18:35:02.458283 18498 catalog_manager.cc:5495] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 016769fe7fa046d9814241fcc84b8022 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "016769fe7fa046d9814241fcc84b8022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016769fe7fa046d9814241fcc84b8022" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } health_report { overall_health: HEALTHY } } }
I20260929 18:35:02.498205 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.494519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.0.98.2:43819)
0929 18:35:02.494652 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.494657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.494676 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:35:02.494765 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.494768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.494820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.494933 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:02.495183 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.495717 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.496655 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.496872 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.496977 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.497332 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:02.497773 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:02.497928 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.498012 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.498047 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:35:02.503943 18644 raft_consensus.cc:1261] T 316cd650c98845f7ae3bf8936cc4e012 P 8deda9dbef4344edb8303f8977d75e73 [term 1 FOLLOWER]: Refusing update from remote peer 016769fe7fa046d9814241fcc84b8022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:02.504213 18912 raft_consensus.cc:1261] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 1 FOLLOWER]: Refusing update from remote peer 016769fe7fa046d9814241fcc84b8022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:02.504731 18982 consensus_queue.cc:1035] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8deda9dbef4344edb8303f8977d75e73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:02.504935 18985 consensus_queue.cc:1035] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a078711758aa4a6199c539ecd4301bd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:02.512996 18989 mvcc.cc:205] Tried to move back new op lower bound from 7334735472649015296 to 7334735472458440704. Current Snapshot: MvccSnapshot[applied={T|T < 7334735472649015296}]
I20260929 18:35:02.514276 18990 mvcc.cc:205] Tried to move back new op lower bound from 7334735472649015296 to 7334735472458440704. Current Snapshot: MvccSnapshot[applied={T|T < 7334735472649015296 or (T in {7334735472649015296})}]
I20260929 18:35:02.522799 18497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:02.522984 18497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:02.525628 18497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c90652220bd491fa0cb644448fb3b7c: Sending DeleteTablet for 3 replicas of tablet 316cd650c98845f7ae3bf8936cc4e012
I20260929 18:35:02.526403 18892 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48472
I20260929 18:35:02.526592 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50200
I20260929 18:35:02.527029 18998 tablet_replica.cc:330] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: stopping tablet replica
I20260929 18:35:02.527393 18998 raft_consensus.cc:2229] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:02.527396 18758 tablet_service.cc:1514] Processing DeleteTablet for tablet 316cd650c98845f7ae3bf8936cc4e012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36770
I20260929 18:35:02.527747 18998 raft_consensus.cc:2258] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:02.528548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18561
I20260929 18:35:02.529498 18998 ts_tablet_manager.cc:1905] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:02.532409 18998 ts_tablet_manager.cc:1918] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:02.532568 18998 log.cc:1196] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/316cd650c98845f7ae3bf8936cc4e012
I20260929 18:35:02.533020 18998 ts_tablet_manager.cc:1939] T 316cd650c98845f7ae3bf8936cc4e012 P a078711758aa4a6199c539ecd4301bd5: Deleting consensus metadata
I20260929 18:35:02.533537 19000 tablet_replica.cc:330] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022: stopping tablet replica
I20260929 18:35:02.533908 19000 raft_consensus.cc:2229] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:02.534308 19000 raft_consensus.cc:2258] T 316cd650c98845f7ae3bf8936cc4e012 P 016769fe7fa046d9814241fcc84b8022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:02.535568 18483 catalog_manager.cc:4841] TS a078711758aa4a6199c539ecd4301bd5 (127.0.98.3:38245): tablet 316cd650c98845f7ae3bf8936cc4e012 (table test-table [id=a2609b84d83b4bc79bb4ba8521d57a1b]) successfully deleted
W20260929 18:35:02.539742 18484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33301 (error 108)
I20260929 18:35:02.539975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18695
I20260929 18:35:02.542122 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.541823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33301 (local address 127.0.0.1:50202)
0929 18:35:02.541948 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.542053 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:35:02.542264 18484 catalog_manager.cc:4584] TS 8deda9dbef4344edb8303f8977d75e73 (127.0.98.1:33301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 316cd650c98845f7ae3bf8936cc4e012: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33301: connect: Connection refused (error 111)
I20260929 18:35:02.545822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18829
I20260929 18:35:02.546567 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.546132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43819 (local address 127.0.0.1:36786)
0929 18:35:02.546413 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:02.546473 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 18:35:02.551350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 18467
2026-09-29T18:35:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2377 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:02Z Disabled control of system clock
I20260929 18:35:02.581812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41195
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:35:02.696190 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:02.696448 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:02.696524 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:02.700510 19005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:02.700560 19005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:02.700575 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:02.700592 19005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:02.700608 19005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:02.705403 19005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:02.706489 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:02.707875 19005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:02.716408 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.716398 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.716398 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:02.716722 19005 server_base.cc:1034] running on GCE node
I20260929 18:35:02.717296 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:02.718441 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:02.719653 19005 hybrid_clock.cc:648] HybridClock initialized: now 1790706902719624 us; error 51 us; skew 500 ppm
I20260929 18:35:02.721657 19005 webserver.cc:460] Webserver started at http://127.0.98.62:39115/ using document root <none> and password file <none>
I20260929 18:35:02.722275 19005 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:02.722368 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:02.722597 19005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:02.724943 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "7925badad2584ccba7a9ff3fa7eb9b77"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "c79512cb774e85d4858dccdba8be5679"
server_key_iv: "cc862cf5f30b02bd676e271db46f952b"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.725535 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "7925badad2584ccba7a9ff3fa7eb9b77"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "c79512cb774e85d4858dccdba8be5679"
server_key_iv: "cc862cf5f30b02bd676e271db46f952b"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.729301 19005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:35:02.731832 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.733153 19005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:02.733296 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "7925badad2584ccba7a9ff3fa7eb9b77"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "c79512cb774e85d4858dccdba8be5679"
server_key_iv: "cc862cf5f30b02bd676e271db46f952b"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.733415 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:02.747632 19005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:02.748401 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:02.748594 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:02.755596 19005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41195
I20260929 18:35:02.755607 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41195 every 8 connection(s)
I20260929 18:35:02.756577 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:02.757400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19005
I20260929 18:35:02.757546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:02.757856 392 external_mini_cluster.cc:1428] Setting key edbf38e15d64affeafa7e6f182947c53
I20260929 18:35:02.760397 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:02.766621 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: Bootstrap starting.
I20260929 18:35:02.768510 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.759338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.62:41195)
0929 18:35:02.759977 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.759992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.760028 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:02.760817 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.760835 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.761228 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.761558 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:02.762001 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.762955 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.763951 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.764750 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.764916 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.766196 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:02.766202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:02.766223 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:02.766817 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:02.766892 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:02.766920 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:02.767018 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.767232 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.767512 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260929 18:35:02.769680 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:02.770443 19072 log.cc:824] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:02.772617 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: No bootstrap required, opened a new log
I20260929 18:35:02.775986 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260929 18:35:02.776266 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:02.776340 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7925badad2584ccba7a9ff3fa7eb9b77, State: Initialized, Role: FOLLOWER
I20260929 18:35:02.776772 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260929 18:35:02.776918 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:02.776986 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:02.777123 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:02.778044 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260929 18:35:02.778393 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7925badad2584ccba7a9ff3fa7eb9b77; no voters:
I20260929 18:35:02.778698 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:02.778847 19077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:02.779122 19077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [term 1 LEADER]: Becoming Leader. State: Replica: 7925badad2584ccba7a9ff3fa7eb9b77, State: Running, Role: LEADER
I20260929 18:35:02.779422 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260929 18:35:02.779928 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:02.781291 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } } }
I20260929 18:35:02.781291 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7925badad2584ccba7a9ff3fa7eb9b77. Latest consensus state: current_term: 1 leader_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7925badad2584ccba7a9ff3fa7eb9b77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } } }
I20260929 18:35:02.781448 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:02.781448 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:02.781898 19087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:02.784832 19087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:02.789769 19087 catalog_manager.cc:1347] Generated new cluster ID: 8ce804e52c9547e5beb406c93b95a69b
I20260929 18:35:02.789832 19087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:02.806288 19087 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:02.807554 19087 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:02.819190 19087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: Generated new TSK 0
I20260929 18:35:02.820029 19087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:02.823069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--builtin_ntp_servers=127.0.98.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:35:02.936757 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:02.937099 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:02.937144 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:02.941195 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:02.941252 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:02.941306 19096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:35:02.945950 19096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:02.946991 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:02.948424 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:02.957960 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.957984 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.958001 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:02.958171 19096 server_base.cc:1034] running on GCE node
I20260929 18:35:02.958753 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:02.959350 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:02.960532 19096 hybrid_clock.cc:648] HybridClock initialized: now 1790706902960498 us; error 50 us; skew 500 ppm
I20260929 18:35:02.962368 19096 webserver.cc:460] Webserver started at http://127.0.98.1:33225/ using document root <none> and password file <none>
I20260929 18:35:02.962944 19096 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:02.963032 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:02.963255 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:02.965330 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4c2fe86a65045ab99a99cd621b9a3c5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "39c0207f3afa206f97e6e32f3ee65d6f"
server_key_iv: "0004c7ea3cffa1291643e3596b8af453"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.965898 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4c2fe86a65045ab99a99cd621b9a3c5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "39c0207f3afa206f97e6e32f3ee65d6f"
server_key_iv: "0004c7ea3cffa1291643e3596b8af453"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.969597 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:02.972025 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.973363 19096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:02.973495 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "f4c2fe86a65045ab99a99cd621b9a3c5"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-fmv8"
server_key: "39c0207f3afa206f97e6e32f3ee65d6f"
server_key_iv: "0004c7ea3cffa1291643e3596b8af453"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.973614 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:02.985558 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:02.986277 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:02.986479 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:02.987053 19096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:02.987994 19096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:02.988062 19096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.988168 19096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:02.988219 19096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.997289 19096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37829
I20260929 18:35:02.997330 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37829 every 8 connection(s)
I20260929 18:35:02.998200 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:02.999135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19096
I20260929 18:35:02.999274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:02.999600 392 external_mini_cluster.cc:1428] Setting key 13ea0a5510d00a45bdccc90514cc7745
I20260929 18:35:03.001892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--builtin_ntp_servers=127.0.98.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:35:03.008482 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.001681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59071 (local address 127.0.98.62:41195)
0929 18:35:03.001796 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.001799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.001812 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:03.003745 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.003751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.003814 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.003911 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:03.004844 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.005426 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.006568 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.006791 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.006899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.007950 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:03.007958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:03.007960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:03.007972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:03.007990 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:03.007998 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:03.008068 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.008252 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.008336 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:35:03.009289 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.000686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.1:59071)
0929 18:35:03.001519 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.001574 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.003529 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.004096 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.004103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.004237 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:03.004718 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.004732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.005557 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.005561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.006446 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.006454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.006934 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.006938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.007106 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.007696 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:03.007698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:03.007735 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:03.007794 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:03.008082 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:03.008093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:03.008098 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.008250 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.008526 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":137,"threads_started":1}
I20260929 18:35:03.010689 19224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260929 18:35:03.010916 19224 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:03.011454 19224 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
I20260929 18:35:03.012941 19036 ts_manager.cc:194] Registered new tserver with Master: f4c2fe86a65045ab99a99cd621b9a3c5 (127.0.98.1:37829)
I20260929 18:35:03.014065 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59071
W20260929 18:35:03.120373 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:03.120718 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:03.120795 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:03.125423 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:03.125515 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:03.125613 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:35:03.130981 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:03.132112 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:03.133571 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:03.143477 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:03.143493 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:03.143491 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:03.143805 19227 server_base.cc:1034] running on GCE node
I20260929 18:35:03.144340 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:03.144991 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:03.146169 19227 hybrid_clock.cc:648] HybridClock initialized: now 1790706903146137 us; error 45 us; skew 500 ppm
I20260929 18:35:03.147998 19227 webserver.cc:460] Webserver started at http://127.0.98.2:40025/ using document root <none> and password file <none>
I20260929 18:35:03.148612 19227 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:03.148675 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:03.148837 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:03.150803 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "c088843a8c44441d8ba5d1c4d0c52380"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "928bc835f5b7b1063a1c42f01b04fe19"
server_key_iv: "4f45308114674c58bd6e50049bcb79c3"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.151309 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c088843a8c44441d8ba5d1c4d0c52380"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "928bc835f5b7b1063a1c42f01b04fe19"
server_key_iv: "4f45308114674c58bd6e50049bcb79c3"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.154989 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 18:35:03.157400 19241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.158612 19227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:03.158732 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "c088843a8c44441d8ba5d1c4d0c52380"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "928bc835f5b7b1063a1c42f01b04fe19"
server_key_iv: "4f45308114674c58bd6e50049bcb79c3"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.158839 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:03.180271 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:03.181078 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:03.181267 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:03.181960 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:03.182930 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:03.183013 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.183089 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:03.183131 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.191996 19227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38503
I20260929 18:35:03.192093 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38503 every 8 connection(s)
I20260929 18:35:03.192965 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:03.199447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19227
I20260929 18:35:03.199595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:03.199919 392 external_mini_cluster.cc:1428] Setting key b8a1e21fdf9d9b2c103668da312ed433
I20260929 18:35:03.202200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--builtin_ntp_servers=127.0.98.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:35:03.202843 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.194851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39567 (local address 127.0.98.62:41195)
0929 18:35:03.195068 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.195072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.196097 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 18:35:03.197532 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.197535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.197596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.197674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:03.198558 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.199090 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.200727 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.201013 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.201119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.202367 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:03.202373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:03.202378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:03.202395 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:03.202423 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:03.202433 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:03.202498 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.202691 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.202740 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:35:03.204034 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.195245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.2:39567)
0929 18:35:03.195896 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.195937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.197342 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.197830 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.197836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.197919 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:03.198438 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.198451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.199235 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.199240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.200605 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.200616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.201155 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.201158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.201381 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.202024 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:03.202027 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:03.202083 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:03.202176 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:03.202526 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:03.202538 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:03.202547 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.202679 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.203036 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260929 18:35:03.205284 19355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260929 18:35:03.205536 19355 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:03.206096 19355 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
I20260929 18:35:03.207108 19036 ts_manager.cc:194] Registered new tserver with Master: c088843a8c44441d8ba5d1c4d0c52380 (127.0.98.2:38503)
I20260929 18:35:03.207682 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39567
W20260929 18:35:03.313189 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:03.313446 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:03.313485 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:03.317575 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:03.317637 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:03.317692 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:35:03.323117 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:03.324314 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:03.325682 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:03.335268 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:03.335254 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:03.335314 19358 server_base.cc:1034] running on GCE node
W20260929 18:35:03.335250 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:03.335853 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:03.336552 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:03.337731 19358 hybrid_clock.cc:648] HybridClock initialized: now 1790706903337721 us; error 33 us; skew 500 ppm
I20260929 18:35:03.339489 19358 webserver.cc:460] Webserver started at http://127.0.98.3:44663/ using document root <none> and password file <none>
I20260929 18:35:03.340083 19358 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:03.340189 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:03.340404 19358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:03.342418 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "baff79da56f4446b941b37d13033ebc6"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "a41ced4cd25cee358c25d4b9694363b4"
server_key_iv: "ebed475f791571f839e5e0ba9dd61836"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.342978 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "baff79da56f4446b941b37d13033ebc6"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "a41ced4cd25cee358c25d4b9694363b4"
server_key_iv: "ebed475f791571f839e5e0ba9dd61836"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.346509 19358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:35:03.348795 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.350103 19358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:35:03.350222 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "baff79da56f4446b941b37d13033ebc6"
format_stamp: "Formatted at 2026-09-29 18:35:03 on dist-test-slave-fmv8"
server_key: "a41ced4cd25cee358c25d4b9694363b4"
server_key_iv: "ebed475f791571f839e5e0ba9dd61836"
server_key_version: "encryptionkey@0"
I20260929 18:35:03.350332 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:03.361971 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:03.362592 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:03.362813 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:03.363384 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:03.364328 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:03.364375 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.364414 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:03.364460 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:03.373606 19358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38467
I20260929 18:35:03.373646 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38467 every 8 connection(s)
I20260929 18:35:03.374536 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:03.378773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19358
I20260929 18:35:03.378888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:03.379220 392 external_mini_cluster.cc:1428] Setting key 8e36c766f876c41fa60ffe934369499e
I20260929 18:35:03.384603 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.376352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57359 (local address 127.0.98.62:41195)
0929 18:35:03.376508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.376511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.377546 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 18:35:03.379207 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.379211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.379259 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.379397 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:03.380561 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.381202 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.382608 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.382827 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.382931 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.384042 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:03.384045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:03.384047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:03.384058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:03.384081 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:03.384089 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:03.384186 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.384381 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.384463 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:35:03.385383 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.376719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.3:57359)
0929 18:35:03.377380 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.377419 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.378967 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.379579 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.379588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.379698 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:03.380415 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.380431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.381313 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.381317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.382496 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.382505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.382972 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.382976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.383148 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.383756 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:03.383757 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:03.383800 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:03.383860 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:03.384227 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:03.384235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:03.384241 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.384369 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.384632 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260929 18:35:03.386514 19486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260929 18:35:03.386744 19486 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:03.387233 19486 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
I20260929 18:35:03.388276 19036 ts_manager.cc:194] Registered new tserver with Master: baff79da56f4446b941b37d13033ebc6 (127.0.98.3:38467)
I20260929 18:35:03.388827 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57359
I20260929 18:35:03.393348 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:03.399047 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.395176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.0.98.62:41195)
0929 18:35:03.395325 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.395329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.395493 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:35:03.395594 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.395597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.395636 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.395709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:03.396035 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.396642 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.397477 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.397681 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.397759 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.398571 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:03.398574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:03.398576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:03.398586 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:03.398604 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:03.398611 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:03.398702 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.398821 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.398878 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:35:03.401944 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:03.404894 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:03.418910 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37829 (local address 127.0.0.1:44878)
0929 18:35:03.413805 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.413859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.413985 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.414786 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.414793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.414834 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:03.414970 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.414977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.416635 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.416641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.417706 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.417715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418548 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.418552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.418649 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.418668 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.418711 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.418756 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":40,"spinlock_wait_cycles":14976,"thread_start_us":91,"threads_started":1}
I20260929 18:35:03.418954 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.98.3:38467)
0929 18:35:03.414271 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.414283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.414307 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:35:03.414384 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.414389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.414625 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.414781 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:03.415080 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.415849 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.417400 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418208 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.418316 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.418503 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.418662 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.418783 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":151,"threads_started":1}
I20260929 18:35:03.419318 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38503 (local address 127.0.0.1:40612)
0929 18:35:03.413663 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.414101 (+ 438us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.414170 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.414786 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.414793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.414836 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:03.415126 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.415130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.416548 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.416551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.417748 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.417757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418799 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.418803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.418896 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.418919 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.418970 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.419207 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":261,"spinlock_wait_cycles":25984,"thread_start_us":77,"threads_started":1}
I20260929 18:35:03.419492 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.98.1:37829)
0929 18:35:03.414407 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.414417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.414435 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:35:03.414507 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.414511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.414652 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.415277 (+ 625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:03.415406 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.416529 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.417820 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418401 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.418979 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.419176 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.419225 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.419313 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":101,"threads_started":1}
I20260929 18:35:03.418910 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.0.1:51414)
0929 18:35:03.413881 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.413895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.414041 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.414798 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.414801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.414834 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:03.414965 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.414970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.415964 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.415968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.417260 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.417271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418331 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.418335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.418452 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.418509 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.418613 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.418735 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"spinlock_wait_cycles":7680,"thread_start_us":107,"threads_started":1}
I20260929 18:35:03.420087 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.413155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.0.98.2:38503)
0929 18:35:03.414297 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.414314 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.414336 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:35:03.414410 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.414416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.414608 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.414923 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:03.415227 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.416423 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.417879 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.418664 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.419631 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.419811 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.419857 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.419941 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":123,"threads_started":1}
I20260929 18:35:03.422279 19420 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f (DEFAULT_TABLE table=test-table [id=f1f6054392b44c369eab79ccb867c9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:03.422415 19289 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f (DEFAULT_TABLE table=test-table [id=f1f6054392b44c369eab79ccb867c9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:03.422415 19158 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f (DEFAULT_TABLE table=test-table [id=f1f6054392b44c369eab79ccb867c9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:03.423252 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0e3f8418dd47acbeaa810463f1e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:03.423377 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0e3f8418dd47acbeaa810463f1e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:03.423377 19158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0e3f8418dd47acbeaa810463f1e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:03.430163 19503 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Bootstrap starting.
I20260929 18:35:03.430687 19505 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Bootstrap starting.
I20260929 18:35:03.430687 19504 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: Bootstrap starting.
I20260929 18:35:03.432353 19503 tablet_bootstrap.cc:654] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:03.432799 19505 tablet_bootstrap.cc:654] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:03.432799 19504 tablet_bootstrap.cc:654] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:03.433108 19503 log.cc:824] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:03.433617 19504 log.cc:824] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:03.433617 19505 log.cc:824] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:03.435302 19503 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: No bootstrap required, opened a new log
I20260929 18:35:03.435500 19503 ts_tablet_manager.cc:1397] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:35:03.436003 19505 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: No bootstrap required, opened a new log
I20260929 18:35:03.436045 19504 tablet_bootstrap.cc:492] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: No bootstrap required, opened a new log
I20260929 18:35:03.436223 19505 ts_tablet_manager.cc:1397] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:03.436223 19504 ts_tablet_manager.cc:1397] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:03.438956 19503 raft_consensus.cc:348] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.439188 19503 raft_consensus.cc:374] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:03.439239 19503 raft_consensus.cc:729] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baff79da56f4446b941b37d13033ebc6, State: Initialized, Role: FOLLOWER
I20260929 18:35:03.439677 19503 consensus_queue.cc:260] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.440181 19504 raft_consensus.cc:348] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.440358 19486 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
I20260929 18:35:03.440440 19504 raft_consensus.cc:374] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:03.440492 19504 raft_consensus.cc:729] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c2fe86a65045ab99a99cd621b9a3c5, State: Initialized, Role: FOLLOWER
I20260929 18:35:03.440701 19503 ts_tablet_manager.cc:1428] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:03.440902 19504 consensus_queue.cc:260] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.440181 19505 raft_consensus.cc:348] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.441160 19505 raft_consensus.cc:374] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:03.441220 19505 raft_consensus.cc:729] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088843a8c44441d8ba5d1c4d0c52380, State: Initialized, Role: FOLLOWER
I20260929 18:35:03.441617 19224 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
I20260929 18:35:03.442203 19504 ts_tablet_manager.cc:1428] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:03.442693 19505 consensus_queue.cc:260] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.443138 19512 raft_consensus.cc:482] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:03.443339 19512 raft_consensus.cc:504] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.443673 19505 ts_tablet_manager.cc:1428] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260929 18:35:03.443979 19355 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
W20260929 18:35:03.443984 19356 tablet.cc:2369] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:03.445755 19512 leader_election.cc:288] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c2fe86a65045ab99a99cd621b9a3c5 (127.0.98.1:37829), baff79da56f4446b941b37d13033ebc6 (127.0.98.3:38467)
I20260929 18:35:03.449621 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.445471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37829 (local address 127.0.98.2:49979)
0929 18:35:03.445592 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.445607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.445685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.446061 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.446064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.446083 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:03.446206 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.446213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.447186 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.447190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.448495 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.448505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.449278 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.449280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.449350 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.449369 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.449426 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.449475 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 18:35:03.449656 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.445717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49979 (local address 127.0.98.1:37829)
0929 18:35:03.445830 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.445836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.445853 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:03.445910 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.445915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.445958 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.446036 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:03.446338 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.447066 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.448613 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.449169 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.449274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.449333 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.449458 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.449546 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:03.450068 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.446461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57087 (local address 127.0.98.3:38467)
0929 18:35:03.446569 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.446575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.446590 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:35:03.446629 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.446632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.446665 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.446746 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:03.447013 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.447583 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.448877 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.449473 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.449569 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.449619 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.449855 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.449941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:35:03.450083 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.446090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.2:57087)
0929 18:35:03.446315 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.446329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:03.446523 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:03.446765 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:03.446767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:03.446783 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:03.446896 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.446903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.447694 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.447697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.448780 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:03.448787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.449591 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.449594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:03.449694 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.449718 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:35:03.449836 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:35:03.449907 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260929 18:35:03.450243 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0e3f8418dd47acbeaa810463f1e58f" candidate_uuid: "c088843a8c44441d8ba5d1c4d0c52380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" is_pre_election: true
I20260929 18:35:03.450526 19178 raft_consensus.cc:2454] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088843a8c44441d8ba5d1c4d0c52380 in term 0.
I20260929 18:35:03.450660 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0e3f8418dd47acbeaa810463f1e58f" candidate_uuid: "c088843a8c44441d8ba5d1c4d0c52380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baff79da56f4446b941b37d13033ebc6" is_pre_election: true
I20260929 18:35:03.450918 19440 raft_consensus.cc:2454] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088843a8c44441d8ba5d1c4d0c52380 in term 0.
I20260929 18:35:03.451020 19243 leader_election.cc:302] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c088843a8c44441d8ba5d1c4d0c52380, f4c2fe86a65045ab99a99cd621b9a3c5; no voters:
I20260929 18:35:03.451265 19512 raft_consensus.cc:2781] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:03.451354 19512 raft_consensus.cc:482] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:03.451404 19512 raft_consensus.cc:3037] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:03.452404 19512 raft_consensus.cc:504] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.452736 19512 leader_election.cc:288] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [CANDIDATE]: Term 1 election: Requested vote from peers f4c2fe86a65045ab99a99cd621b9a3c5 (127.0.98.1:37829), baff79da56f4446b941b37d13033ebc6 (127.0.98.3:38467)
I20260929 18:35:03.453191 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0e3f8418dd47acbeaa810463f1e58f" candidate_uuid: "c088843a8c44441d8ba5d1c4d0c52380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baff79da56f4446b941b37d13033ebc6"
I20260929 18:35:03.453246 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0e3f8418dd47acbeaa810463f1e58f" candidate_uuid: "c088843a8c44441d8ba5d1c4d0c52380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5"
I20260929 18:35:03.453315 19440 raft_consensus.cc:3037] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:03.453393 19178 raft_consensus.cc:3037] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:03.454340 19440 raft_consensus.cc:2454] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088843a8c44441d8ba5d1c4d0c52380 in term 1.
I20260929 18:35:03.454576 19178 raft_consensus.cc:2454] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088843a8c44441d8ba5d1c4d0c52380 in term 1.
I20260929 18:35:03.454625 19242 leader_election.cc:302] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baff79da56f4446b941b37d13033ebc6, c088843a8c44441d8ba5d1c4d0c52380; no voters:
I20260929 18:35:03.454789 19512 raft_consensus.cc:2781] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:03.454901 19512 raft_consensus.cc:686] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 1 LEADER]: Becoming Leader. State: Replica: c088843a8c44441d8ba5d1c4d0c52380, State: Running, Role: LEADER
I20260929 18:35:03.455219 19512 consensus_queue.cc:237] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260929 18:35:03.458104 19036 catalog_manager.cc:5495] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 reported cstate change: term changed from 0 to 1, leader changed from <none> to c088843a8c44441d8ba5d1c4d0c52380 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c088843a8c44441d8ba5d1c4d0c52380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c088843a8c44441d8ba5d1c4d0c52380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:03.468771 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.465344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.98.2:38503)
0929 18:35:03.465474 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:03.465478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:03.465491 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:03.465545 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:03.465547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:03.465593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:03.465685 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:03.465919 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.466442 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.467309 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:03.467524 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:03.467630 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:03.467838 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:03.468388 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:03.468512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:03.468622 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:35:03.468654 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:35:03.474532 19178 raft_consensus.cc:1261] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 1 FOLLOWER]: Refusing update from remote peer c088843a8c44441d8ba5d1c4d0c52380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:03.474532 19440 raft_consensus.cc:1261] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 1 FOLLOWER]: Refusing update from remote peer c088843a8c44441d8ba5d1c4d0c52380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:03.475385 19512 consensus_queue.cc:1035] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baff79da56f4446b941b37d13033ebc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:03.475660 19511 consensus_queue.cc:1035] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c2fe86a65045ab99a99cd621b9a3c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:03.483918 19517 mvcc.cc:205] Tried to move back new op lower bound from 7334735476624404480 to 7334735476555198464. Current Snapshot: MvccSnapshot[applied={T|T < 7334735476624404480}]
I20260929 18:35:03.492200 19035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:03.492383 19035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:03.494988 19035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7925badad2584ccba7a9ff3fa7eb9b77: Sending DeleteTablet for 3 replicas of tablet 4c0e3f8418dd47acbeaa810463f1e58f
I20260929 18:35:03.495795 19158 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:03 UTC) from {username='slave'} at 127.0.0.1:44878
I20260929 18:35:03.495942 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:03 UTC) from {username='slave'} at 127.0.0.1:51414
I20260929 18:35:03.495942 19289 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0e3f8418dd47acbeaa810463f1e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:03 UTC) from {username='slave'} at 127.0.0.1:40612
I20260929 18:35:03.496330 19526 tablet_replica.cc:330] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5: stopping tablet replica
I20260929 18:35:03.496459 19527 tablet_replica.cc:330] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: stopping tablet replica
I20260929 18:35:03.496764 19527 raft_consensus.cc:2229] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:03.496764 19526 raft_consensus.cc:2229] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:03.496901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19096
I20260929 18:35:03.497092 19527 raft_consensus.cc:2258] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:03.497196 19526 raft_consensus.cc:2258] T 4c0e3f8418dd47acbeaa810463f1e58f P f4c2fe86a65045ab99a99cd621b9a3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:03.499191 19527 ts_tablet_manager.cc:1905] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:03.500392 19528 tablet_replica.cc:330] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: stopping tablet replica
I20260929 18:35:03.500766 19528 raft_consensus.cc:2229] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:03.501286 19528 raft_consensus.cc:2258] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:03.502207 19527 ts_tablet_manager.cc:1918] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:03.502315 19527 log.cc:1196] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/4c0e3f8418dd47acbeaa810463f1e58f
I20260929 18:35:03.502677 19527 ts_tablet_manager.cc:1939] T 4c0e3f8418dd47acbeaa810463f1e58f P baff79da56f4446b941b37d13033ebc6: Deleting consensus metadata
I20260929 18:35:03.503165 19528 ts_tablet_manager.cc:1905] T 4c0e3f8418dd47acbeaa810463f1e58f P c088843a8c44441d8ba5d1c4d0c52380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:03.503968 19020 catalog_manager.cc:4841] TS baff79da56f4446b941b37d13033ebc6 (127.0.98.3:38467): tablet 4c0e3f8418dd47acbeaa810463f1e58f (table test-table [id=f1f6054392b44c369eab79ccb867c9e0]) successfully deleted
W20260929 18:35:03.504853 19021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37829 (error 108)
I20260929 18:35:03.505657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19227
I20260929 18:35:03.505996 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.505609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37829 (local address 127.0.0.1:44890)
0929 18:35:03.505797 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.505880 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260929 18:35:03.506341 19021 catalog_manager.cc:4584] TS f4c2fe86a65045ab99a99cd621b9a3c5 (127.0.98.1:37829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c0e3f8418dd47acbeaa810463f1e58f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37829: connect: Connection refused (error 111)
I20260929 18:35:03.511273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19358
I20260929 18:35:03.512459 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:03.511453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38503 (local address 127.0.0.1:40634)
0929 18:35:03.512298 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:03.512361 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":746}
I20260929 18:35:03.516916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19005
2026-09-29T18:35:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:03 dist-test-slave-fmv8 krb5kdc[19532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:03 dist-test-slave-fmv8 krb5kdc[19532](info): set up 2 sockets
Sep 29 18:35:03 dist-test-slave-fmv8 krb5kdc[19532](info): commencing operation
krb5kdc: starting...
W20260929 18:35:04.568109 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:04 dist-test-slave-fmv8 krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:04.693370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:04.807931 19545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:04.808228 19545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:04.808269 19545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:04.812335 19545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:04.812395 19545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:04.812415 19545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:04.812433 19545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:04.812455 19545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:04.817643 19545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:04.818710 19545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:04.819710 19545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:04.828090 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.828073 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.828089 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.828467 19545 server_base.cc:1034] running on GCE node
I20260929 18:35:04.829042 19545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:04.830073 19545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:04.831234 19545 hybrid_clock.cc:648] HybridClock initialized: now 1790706904831205 us; error 43 us; skew 500 ppm
Sep 29 18:35:04 dist-test-slave-fmv8 krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:04.833935 19545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:04.834856 19545 webserver.cc:460] Webserver started at http://127.0.98.62:38801/ using document root <none> and password file <none>
I20260929 18:35:04.835420 19545 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:04.835531 19545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:04.835886 19545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:04.838078 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "e166fc0bb88d436d8752d7ac8f59d76d"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-fmv8"
server_key: "d5ac1b0b821027d3be531c0ece9ccb9d"
server_key_iv: "0ee90a4da7eb89bc981b563a55212079"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.838670 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "e166fc0bb88d436d8752d7ac8f59d76d"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-fmv8"
server_key: "d5ac1b0b821027d3be531c0ece9ccb9d"
server_key_iv: "0ee90a4da7eb89bc981b563a55212079"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.842648 19545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:04.845381 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.846813 19545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:04.847029 19545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "e166fc0bb88d436d8752d7ac8f59d76d"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-fmv8"
server_key: "d5ac1b0b821027d3be531c0ece9ccb9d"
server_key_iv: "0ee90a4da7eb89bc981b563a55212079"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.847160 19545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:04.857831 19545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:04.861541 19545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:04.861737 19545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:04.869918 19545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37689
I20260929 18:35:04.869935 19612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37689 every 8 connection(s)
I20260929 18:35:04.871014 19545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:04.872027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19545
I20260929 18:35:04.872187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:04.872483 392 external_mini_cluster.cc:1428] Setting key ff863121a83a0df994793624e4b6e1b7
I20260929 18:35:04.874101 19613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:04.880827 19613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: Bootstrap starting.
Sep 29 18:35:04 dist-test-slave-fmv8 krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:04.883313 19613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:04.884032 19613 log.cc:824] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:04.886272 19613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: No bootstrap required, opened a new log
I20260929 18:35:04.887542 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.873894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.98.62:37689)
0929 18:35:04.874313 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:04.874332 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:04.874371 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:35:04.875185 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:04.875206 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:04.875608 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:04.875961 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:04.876723 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.878243 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.879105 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.879790 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.879971 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.882698 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:04.882702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:04.882715 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:04.882742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:04.884711 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.885193 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.885199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.885203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.885307 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.885584 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.885586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.885587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.886014 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:04.886188 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:04.886487 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:35:04.886794 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260929 18:35:04.890092 19613 raft_consensus.cc:348] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260929 18:35:04.890302 19613 raft_consensus.cc:374] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:04.890390 19613 raft_consensus.cc:729] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e166fc0bb88d436d8752d7ac8f59d76d, State: Initialized, Role: FOLLOWER
I20260929 18:35:04.890815 19613 consensus_queue.cc:260] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260929 18:35:04.890959 19613 raft_consensus.cc:388] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:04.891040 19613 raft_consensus.cc:482] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:04.891147 19613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:04.892107 19613 raft_consensus.cc:504] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260929 18:35:04.892495 19613 leader_election.cc:302] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e166fc0bb88d436d8752d7ac8f59d76d; no voters:
I20260929 18:35:04.892796 19613 leader_election.cc:288] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:04.892915 19618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:04.893148 19618 raft_consensus.cc:686] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [term 1 LEADER]: Becoming Leader. State: Replica: e166fc0bb88d436d8752d7ac8f59d76d, State: Running, Role: LEADER
I20260929 18:35:04.893494 19618 consensus_queue.cc:237] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } }
I20260929 18:35:04.894068 19613 sys_catalog.cc:564] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:04.895243 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } } }
I20260929 18:35:04.895241 19620 sys_catalog.cc:455] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e166fc0bb88d436d8752d7ac8f59d76d. Latest consensus state: current_term: 1 leader_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166fc0bb88d436d8752d7ac8f59d76d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37689 } } }
I20260929 18:35:04.895390 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:04.895388 19620 sys_catalog.cc:458] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:04.896246 19627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:04.898808 19627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:04.904266 19627 catalog_manager.cc:1347] Generated new cluster ID: 5774527282b349bda0400e383ede866a
I20260929 18:35:04.904343 19627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:04.924248 19627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:04.925261 19627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:04.933586 19627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: Generated new TSK 0
I20260929 18:35:04.934337 19627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 18:35:04.967634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:05.078485 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.078790 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.078871 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.082988 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.083074 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.083165 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:35:05.088660 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:05.089717 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.090752 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.100473 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.100556 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.100471 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.100723 19639 server_base.cc:1034] running on GCE node
I20260929 18:35:05.101202 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.101817 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.102988 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790706905102970 us; error 29 us; skew 500 ppm
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:05.105710 19639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 18:35:05.106534 19639 webserver.cc:460] Webserver started at http://127.0.98.1:44143/ using document root <none> and password file <none>
I20260929 18:35:05.107082 19639 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.107167 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.107391 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:05.109423 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4f57aae48d4459681308c1acab67a74"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "8f344cb7a65966d97f7b4e279fbf22a6"
server_key_iv: "1956ad57a0e4fd5b5672543de6a69417"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.109957 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4f57aae48d4459681308c1acab67a74"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "8f344cb7a65966d97f7b4e279fbf22a6"
server_key_iv: "1956ad57a0e4fd5b5672543de6a69417"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.113601 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:05.115938 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.117497 19639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:05.117615 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "b4f57aae48d4459681308c1acab67a74"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "8f344cb7a65966d97f7b4e279fbf22a6"
server_key_iv: "1956ad57a0e4fd5b5672543de6a69417"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.117723 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.136910 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.140236 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.140442 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.141059 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:05.141891 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:05.141943 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.142001 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:05.142032 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.150561 19639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42335
I20260929 18:35:05.150611 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42335 every 8 connection(s)
I20260929 18:35:05.151506 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:05.154330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19639
I20260929 18:35:05.154490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:05.154814 392 external_mini_cluster.cc:1428] Setting key a51e669d8c734cf35551640db595088c
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 18:35:05.166086 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.153294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33881 (local address 127.0.98.62:37689)
0929 18:35:05.153457 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.153462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.154324 (+ 862us) server_negotiation.cc:407] Connection header received
0929 18:35:05.155397 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.155400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.155451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.155547 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.156981 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.157604 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.159180 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.159622 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.159754 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.162165 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.162170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.162175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.162203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.163949 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.164481 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.164485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.164487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.164574 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.164946 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.164950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.164951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.165191 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.165274 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.165843 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.165955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:35:05.166883 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.153692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.1:33881)
0929 18:35:05.154165 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.154206 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.155126 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.155757 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.155766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.156309 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.156824 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.156838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.157767 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.157771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.159065 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.159075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.159771 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.159774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.159947 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.160592 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.160608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.161991 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.164060 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.164065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.164076 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.164387 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.164671 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.164673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.164675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.164831 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.165278 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.165283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.165523 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.165685 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.166000 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 18:35:05.168453 19768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260929 18:35:05.168712 19768 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:05.169381 19768 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260929 18:35:05.171005 19573 ts_manager.cc:194] Registered new tserver with Master: b4f57aae48d4459681308c1acab67a74 (127.0.98.1:42335)
I20260929 18:35:05.172251 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 18:35:05.183146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:05.292750 19773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.293025 19773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.293068 19773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.297477 19773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.297549 19773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.297608 19773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:35:05.303086 19773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:05.304245 19773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.305255 19773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.314729 19779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.314908 19781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.314741 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.315146 19773 server_base.cc:1034] running on GCE node
I20260929 18:35:05.315567 19773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.316288 19773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.317461 19773 hybrid_clock.cc:648] HybridClock initialized: now 1790706905317444 us; error 31 us; skew 500 ppm
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:05.320253 19773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 18:35:05.321179 19773 webserver.cc:460] Webserver started at http://127.0.98.2:45959/ using document root <none> and password file <none>
I20260929 18:35:05.321823 19773 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.321889 19773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.322110 19773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:05.324213 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "0ae11695e69ae1a7ea0faef270c70cf2"
server_key_iv: "7f321bfa5c75e8182f3c8e5fad5d6c59"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.324779 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "0ae11695e69ae1a7ea0faef270c70cf2"
server_key_iv: "7f321bfa5c75e8182f3c8e5fad5d6c59"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.328472 19773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:05.330876 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.332051 19773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:05.332206 19773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "0ae11695e69ae1a7ea0faef270c70cf2"
server_key_iv: "7f321bfa5c75e8182f3c8e5fad5d6c59"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.332315 19773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.354797 19773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.358137 19773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.358341 19773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.358932 19773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:05.359789 19773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:05.359835 19773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.359901 19773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:05.359937 19773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.368674 19773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38221
I20260929 18:35:05.368793 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38221 every 8 connection(s)
I20260929 18:35:05.369620 19773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:05.379437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19773
I20260929 18:35:05.379583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:05.379953 392 external_mini_cluster.cc:1428] Setting key 20cb3cbfccb0cb8dc02584d85aed26d8
I20260929 18:35:05.383129 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.371503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42653 (local address 127.0.98.62:37689)
0929 18:35:05.371632 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.371637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.372482 (+ 845us) server_negotiation.cc:407] Connection header received
0929 18:35:05.373391 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.373395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.373450 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.373536 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.374671 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.375208 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.376377 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.376599 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.376706 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.379102 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.379105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.379107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.379131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.380844 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.381399 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.381403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.381405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.381507 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.381856 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.381860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.381862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.382113 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.382248 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.382689 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.382857 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:05.383649 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.371861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.2:42653)
0929 18:35:05.372343 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.372387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.373183 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.373716 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.373723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.374071 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.374532 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.374544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.375328 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.375332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.376259 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.376269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.376745 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.376749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.376918 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.377450 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.377466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.378933 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.380944 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.380951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.380964 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.381287 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.381625 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.381627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.381629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.381756 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.382218 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.382223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.382451 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.382600 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.382835 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260929 18:35:05.385166 19902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260929 18:35:05.385440 19902 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:05.386050 19902 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260929 18:35:05.387264 19573 ts_manager.cc:194] Registered new tserver with Master: 1c2fbf7c3d9845d3b1dcdf1c277571c3 (127.0.98.2:38221)
I20260929 18:35:05.388020 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42653
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 18:35:05.409694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:05.522984 19907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.523255 19907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.523332 19907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.527366 19907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.527428 19907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.527486 19907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:35:05.532763 19907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:05.533790 19907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.534807 19907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.544265 19915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.544363 19913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.544322 19912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.544528 19907 server_base.cc:1034] running on GCE node
I20260929 18:35:05.544943 19907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.545558 19907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.546722 19907 hybrid_clock.cc:648] HybridClock initialized: now 1790706905546705 us; error 36 us; skew 500 ppm
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:05.549484 19907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 18:35:05.550436 19907 webserver.cc:460] Webserver started at http://127.0.98.3:35283/ using document root <none> and password file <none>
I20260929 18:35:05.550987 19907 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.551045 19907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.551201 19907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:05.553236 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "19e937b6014a1663b14194094cfcd617"
server_key_iv: "510e0dc83bece44d23b58a1ce199a95c"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.553723 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "19e937b6014a1663b14194094cfcd617"
server_key_iv: "510e0dc83bece44d23b58a1ce199a95c"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.557224 19907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:05.559523 19922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.560758 19907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:05.560882 19907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "19e937b6014a1663b14194094cfcd617"
server_key_iv: "510e0dc83bece44d23b58a1ce199a95c"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.560986 19907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.583348 19907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.586768 19907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.586984 19907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.587569 19907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:05.588547 19907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:05.588594 19907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.588657 19907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:05.588690 19907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.597442 19907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33099
I20260929 18:35:05.597517 20035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33099 every 8 connection(s)
I20260929 18:35:05.598366 19907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:05.606181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 19907
I20260929 18:35:05.606324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:05.606617 392 external_mini_cluster.cc:1428] Setting key 33c31d9c2b603c499b6bbe2366d6fc3d
Sep 29 18:35:05 dist-test-slave-fmv8 krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:05.612242 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.600215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55077 (local address 127.0.98.62:37689)
0929 18:35:05.600351 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.600355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.601374 (+ 1019us) server_negotiation.cc:407] Connection header received
0929 18:35:05.602348 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.602352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.602410 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.602507 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.603650 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.604234 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.605396 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.605607 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.605688 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.608913 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.608916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.608919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.608946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.610209 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.610724 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.610731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.610735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.610820 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.611150 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.611154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.611156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.611346 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.611452 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.611937 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.612062 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:35:05.613050 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.600562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37689 (local address 127.0.98.3:55077)
0929 18:35:05.601223 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.601264 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.602140 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.602676 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.602682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.603042 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.603508 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.603521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.604376 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.604383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.605279 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.605289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.605733 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.605736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.605903 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.606522 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.606538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.608735 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.610327 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.610334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.610350 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.610613 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.610914 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.610917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.610919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.611058 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.611448 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.611455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.611702 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.611898 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.612244 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":116,"threads_started":1}
I20260929 18:35:05.614166 20036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37689
I20260929 18:35:05.614531 20036 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:05.615093 20036 heartbeater.cc:511] Master 127.0.98.62:37689 requested a full tablet report, sending...
I20260929 18:35:05.616253 19573 ts_manager.cc:194] Registered new tserver with Master: cfa2fa67c64b42fda75bcb60c7ef6ddd (127.0.98.3:33099)
I20260929 18:35:05.616849 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55077
I20260929 18:35:05.620754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:05.630355 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.622732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.98.62:37689)
0929 18:35:05.622850 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.622860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.623090 (+ 230us) server_negotiation.cc:407] Connection header received
0929 18:35:05.623184 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.623186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.623226 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.623290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.623967 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.624552 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.625570 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.625854 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.625948 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.627376 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.627379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.627381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.627412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.628854 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.629243 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.629245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.629246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.629318 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.629563 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.629565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.629566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.629758 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.629883 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.630091 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.630180 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:05.633484 19573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:05.636679 19573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:05.651437 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.645322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38221 (local address 127.0.0.1:59822)
0929 18:35:05.646559 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.646603 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.646738 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.647267 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.647272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.647293 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.647434 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.647440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.648572 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.648579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.649892 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.649901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.650864 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.650868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.650976 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.651017 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.651164 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.651285 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":91,"threads_started":1}
I20260929 18:35:05.651465 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.645477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.98.2:38221)
0929 18:35:05.646190 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.646203 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.646686 (+ 483us) server_negotiation.cc:407] Connection header received
0929 18:35:05.646909 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.646913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.647107 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.647261 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.647569 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.648454 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.650009 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.650734 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.650843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.651003 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.651207 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.651308 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":117,"threads_started":1}
I20260929 18:35:05.652253 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.645387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42335 (local address 127.0.0.1:45424)
0929 18:35:05.646551 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.646601 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.646739 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.647439 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.647442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.647458 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.647583 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.647590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.649364 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.649367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.650504 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.650513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.651805 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.651809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.651900 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.651916 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.652006 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.652078 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"spinlock_wait_cycles":5760,"thread_start_us":102,"threads_started":1}
I20260929 18:35:05.652812 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.645625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.0.1:33024)
0929 18:35:05.646963 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.646975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.647059 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.647468 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.647471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.647499 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.647615 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.647622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.649147 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.649150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.650454 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.650465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.652225 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.652229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.652510 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.652534 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.652606 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.652671 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260929 18:35:05.652853 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.646012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.0.98.1:42335)
0929 18:35:05.646378 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.646392 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.646704 (+ 312us) server_negotiation.cc:407] Connection header received
0929 18:35:05.646899 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.646907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.647113 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.647276 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.647697 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.649219 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.650623 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.651499 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.651605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.651746 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.652634 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.652731 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":89,"threads_started":1}
I20260929 18:35:05.653457 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.646145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.0.98.3:33099)
0929 18:35:05.646571 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.646588 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.647062 (+ 474us) server_negotiation.cc:407] Connection header received
0929 18:35:05.647163 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.647171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.647336 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.647488 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.647751 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.648991 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.650576 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.651428 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.651990 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.652118 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.653225 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.653318 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":146,"threads_started":1}
I20260929 18:35:05.654589 19836 tablet_service.cc:1467] Processing CreateTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 (DEFAULT_TABLE table=test-table [id=ab0574edf04e485687e6b93e568b0425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.655114 19702 tablet_service.cc:1467] Processing CreateTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 (DEFAULT_TABLE table=test-table [id=ab0574edf04e485687e6b93e568b0425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.655684 19836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07447cd2b4f6441eb598e1d72900ac12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.655910 19702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07447cd2b4f6441eb598e1d72900ac12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.655920 19970 tablet_service.cc:1467] Processing CreateTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 (DEFAULT_TABLE table=test-table [id=ab0574edf04e485687e6b93e568b0425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.656848 19970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07447cd2b4f6441eb598e1d72900ac12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.661944 20053 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: Bootstrap starting.
I20260929 18:35:05.663302 20054 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Bootstrap starting.
I20260929 18:35:05.664556 20053 tablet_bootstrap.cc:654] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.665520 20054 tablet_bootstrap.cc:654] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.665881 20053 log.cc:824] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.666705 20054 log.cc:824] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.667209 20055 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Bootstrap starting.
I20260929 18:35:05.668229 20053 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: No bootstrap required, opened a new log
I20260929 18:35:05.668442 20053 ts_tablet_manager.cc:1397] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:35:05.668794 20054 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: No bootstrap required, opened a new log
I20260929 18:35:05.668974 20054 ts_tablet_manager.cc:1397] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 18:35:05.669400 20055 tablet_bootstrap.cc:654] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.670583 20055 log.cc:824] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.671633 20053 raft_consensus.cc:348] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.671873 20053 raft_consensus.cc:374] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.671923 20053 raft_consensus.cc:729] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f57aae48d4459681308c1acab67a74, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.672381 20053 consensus_queue.cc:260] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.672593 20055 tablet_bootstrap.cc:492] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: No bootstrap required, opened a new log
I20260929 18:35:05.672761 20055 ts_tablet_manager.cc:1397] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:05.672772 20054 raft_consensus.cc:348] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.673038 20054 raft_consensus.cc:374] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.673084 20054 raft_consensus.cc:729] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2fbf7c3d9845d3b1dcdf1c277571c3, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.673353 20053 ts_tablet_manager.cc:1428] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:35:05.673532 20054 consensus_queue.cc:260] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.673872 19768 heartbeater.cc:503] Master 127.0.98.62:37689 was elected leader, sending a full tablet report...
I20260929 18:35:05.675509 20054 ts_tablet_manager.cc:1428] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 18:35:05.676337 19902 heartbeater.cc:503] Master 127.0.98.62:37689 was elected leader, sending a full tablet report...
I20260929 18:35:05.676371 20055 raft_consensus.cc:348] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.676647 20055 raft_consensus.cc:374] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.676708 20055 raft_consensus.cc:729] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa2fa67c64b42fda75bcb60c7ef6ddd, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.677105 20055 consensus_queue.cc:260] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.678042 20055 ts_tablet_manager.cc:1428] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:05.678359 20036 heartbeater.cc:503] Master 127.0.98.62:37689 was elected leader, sending a full tablet report...
I20260929 18:35:05.729923 20059 raft_consensus.cc:482] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:05.730182 20059 raft_consensus.cc:504] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.731367 20059 leader_election.cc:288] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2fbf7c3d9845d3b1dcdf1c277571c3 (127.0.98.2:38221), cfa2fa67c64b42fda75bcb60c7ef6ddd (127.0.98.3:33099)
I20260929 18:35:05.735427 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.731871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58541 (local address 127.0.98.3:33099)
0929 18:35:05.732069 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.732074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.732331 (+ 257us) server_negotiation.cc:407] Connection header received
0929 18:35:05.732481 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.732484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.732530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.732611 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.732917 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.733415 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.734449 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.734848 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.734962 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.735026 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.735225 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.735291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:35:05.735471 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.731783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53883 (local address 127.0.98.2:38221)
0929 18:35:05.731954 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.731960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.732224 (+ 264us) server_negotiation.cc:407] Connection header received
0929 18:35:05.732362 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.732365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.732413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.732496 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.732810 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.733329 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.734427 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.734836 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.734985 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.735047 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.735258 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.735328 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:35:05.735492 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.731675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38221 (local address 127.0.98.1:53883)
0929 18:35:05.732083 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.732096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.732265 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.732506 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.732509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.732524 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.732700 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.732706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.733434 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.733441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.734295 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.734304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.735019 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.735023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.735131 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.735156 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.735243 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.735311 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
I20260929 18:35:05.735526 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.731680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.98.1:58541)
0929 18:35:05.732243 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.732255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.732373 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.732621 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.732624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.732639 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.732811 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.732816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.733524 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.733529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.734319 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.734328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.734977 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.734981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.735078 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.735101 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.735200 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.735274 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":57,"thread_start_us":104,"threads_started":1}
I20260929 18:35:05.736277 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07447cd2b4f6441eb598e1d72900ac12" candidate_uuid: "b4f57aae48d4459681308c1acab67a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" is_pre_election: true
I20260929 18:35:05.736277 19990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07447cd2b4f6441eb598e1d72900ac12" candidate_uuid: "b4f57aae48d4459681308c1acab67a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" is_pre_election: true
I20260929 18:35:05.736614 19990 raft_consensus.cc:2454] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f57aae48d4459681308c1acab67a74 in term 0.
I20260929 18:35:05.736614 19856 raft_consensus.cc:2454] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4f57aae48d4459681308c1acab67a74 in term 0.
I20260929 18:35:05.737160 19655 leader_election.cc:302] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c2fbf7c3d9845d3b1dcdf1c277571c3, b4f57aae48d4459681308c1acab67a74, cfa2fa67c64b42fda75bcb60c7ef6ddd; no voters:
I20260929 18:35:05.737433 20059 raft_consensus.cc:2781] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:05.737520 20059 raft_consensus.cc:482] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:05.737579 20059 raft_consensus.cc:3037] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.738647 20059 raft_consensus.cc:504] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.739091 20059 leader_election.cc:288] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2fbf7c3d9845d3b1dcdf1c277571c3 (127.0.98.2:38221), cfa2fa67c64b42fda75bcb60c7ef6ddd (127.0.98.3:33099)
I20260929 18:35:05.739467 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07447cd2b4f6441eb598e1d72900ac12" candidate_uuid: "b4f57aae48d4459681308c1acab67a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3"
I20260929 18:35:05.739502 19990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07447cd2b4f6441eb598e1d72900ac12" candidate_uuid: "b4f57aae48d4459681308c1acab67a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd"
I20260929 18:35:05.739621 19990 raft_consensus.cc:3037] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.739599 19856 raft_consensus.cc:3037] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.740955 19990 raft_consensus.cc:2454] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f57aae48d4459681308c1acab67a74 in term 1.
I20260929 18:35:05.741088 19856 raft_consensus.cc:2454] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f57aae48d4459681308c1acab67a74 in term 1.
I20260929 18:35:05.741333 19655 leader_election.cc:302] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4f57aae48d4459681308c1acab67a74, cfa2fa67c64b42fda75bcb60c7ef6ddd; no voters:
I20260929 18:35:05.741541 20059 raft_consensus.cc:2781] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:05.741737 20059 raft_consensus.cc:686] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [term 1 LEADER]: Becoming Leader. State: Replica: b4f57aae48d4459681308c1acab67a74, State: Running, Role: LEADER
I20260929 18:35:05.742051 20059 consensus_queue.cc:237] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260929 18:35:05.745237 19577 catalog_manager.cc:5495] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4f57aae48d4459681308c1acab67a74 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b4f57aae48d4459681308c1acab67a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f57aae48d4459681308c1acab67a74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:05.773499 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.769926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.98.1:42335)
0929 18:35:05.770081 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.770086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.770103 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:05.770204 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.770207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.770261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.770381 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:05.770626 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.771157 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.772019 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.772264 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.772383 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.772691 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:05.773112 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:05.773248 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.773284 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.773353 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:35:05.780316 19990 raft_consensus.cc:1261] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 1 FOLLOWER]: Refusing update from remote peer b4f57aae48d4459681308c1acab67a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:05.780304 19856 raft_consensus.cc:1261] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 1 FOLLOWER]: Refusing update from remote peer b4f57aae48d4459681308c1acab67a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:05.780992 20064 consensus_queue.cc:1035] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2fbf7c3d9845d3b1dcdf1c277571c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:05.781147 20059 consensus_queue.cc:1035] T 07447cd2b4f6441eb598e1d72900ac12 P b4f57aae48d4459681308c1acab67a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa2fa67c64b42fda75bcb60c7ef6ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:05.789942 20068 mvcc.cc:205] Tried to move back new op lower bound from 7334735486068686848 to 7334735485922332672. Current Snapshot: MvccSnapshot[applied={T|T < 7334735486068686848}]
I20260929 18:35:05.797083 19577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:05.797279 19577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:05.800061 19577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e166fc0bb88d436d8752d7ac8f59d76d: Sending DeleteTablet for 3 replicas of tablet 07447cd2b4f6441eb598e1d72900ac12
I20260929 18:35:05.800920 19702 tablet_service.cc:1514] Processing DeleteTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45424
I20260929 18:35:05.800933 19836 tablet_service.cc:1514] Processing DeleteTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59822
I20260929 18:35:05.801645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19639
I20260929 18:35:05.801940 19970 tablet_service.cc:1514] Processing DeleteTablet for tablet 07447cd2b4f6441eb598e1d72900ac12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260929 18:35:05.802398 20079 tablet_replica.cc:330] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: stopping tablet replica
I20260929 18:35:05.802695 20079 raft_consensus.cc:2229] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:05.802949 20079 raft_consensus.cc:2258] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:05.803977 20077 tablet_replica.cc:330] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: stopping tablet replica
I20260929 18:35:05.804263 20079 ts_tablet_manager.cc:1905] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:05.804335 20077 raft_consensus.cc:2229] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:05.804651 20077 raft_consensus.cc:2258] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:05.806177 20077 ts_tablet_manager.cc:1905] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:05.806571 20079 ts_tablet_manager.cc:1918] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:05.806679 20079 log.cc:1196] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/07447cd2b4f6441eb598e1d72900ac12
I20260929 18:35:05.806982 20079 ts_tablet_manager.cc:1939] T 07447cd2b4f6441eb598e1d72900ac12 P cfa2fa67c64b42fda75bcb60c7ef6ddd: Deleting consensus metadata
I20260929 18:35:05.808043 19561 catalog_manager.cc:4841] TS cfa2fa67c64b42fda75bcb60c7ef6ddd (127.0.98.3:33099): tablet 07447cd2b4f6441eb598e1d72900ac12 (table test-table [id=ab0574edf04e485687e6b93e568b0425]) successfully deleted
I20260929 18:35:05.809571 20077 ts_tablet_manager.cc:1918] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:05.809671 20077 log.cc:1196] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/07447cd2b4f6441eb598e1d72900ac12
I20260929 18:35:05.810026 20077 ts_tablet_manager.cc:1939] T 07447cd2b4f6441eb598e1d72900ac12 P 1c2fbf7c3d9845d3b1dcdf1c277571c3: Deleting consensus metadata
W20260929 18:35:05.810057 19562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42335 (error 108)
I20260929 18:35:05.810274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19773
I20260929 18:35:05.811591 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.811287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42335 (local address 127.0.0.1:45440)
0929 18:35:05.811451 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.811517 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 18:35:05.811784 19562 catalog_manager.cc:4584] TS b4f57aae48d4459681308c1acab67a74 (127.0.98.1:42335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07447cd2b4f6441eb598e1d72900ac12: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42335: connect: Connection refused (error 111)
I20260929 18:35:05.817016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19907
I20260929 18:35:05.818742 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.817160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38221 (local address 127.0.0.1:59830)
0929 18:35:05.818582 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.818656 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1306}
I20260929 18:35:05.822665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 19545
2026-09-29T18:35:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2307 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:05Z Disabled control of system clock
I20260929 18:35:05.853444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37201
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:35:05.965220 20084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.965500 20084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.965538 20084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.969624 20084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:05.969681 20084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.969697 20084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.969714 20084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:05.969729 20084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:05.974813 20084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:05.975976 20084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.977823 20084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.987407 20092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.987434 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.987434 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.987668 20084 server_base.cc:1034] running on GCE node
I20260929 18:35:05.988487 20084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.989727 20084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.990942 20084 hybrid_clock.cc:648] HybridClock initialized: now 1790706905990907 us; error 45 us; skew 500 ppm
I20260929 18:35:05.993017 20084 webserver.cc:460] Webserver started at http://127.0.98.62:42405/ using document root <none> and password file <none>
I20260929 18:35:05.993570 20084 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.993631 20084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.993798 20084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:05.995865 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "4a31a6940de7485d88921050d690261f"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "7178a95e5643b1531425f420073879b2"
server_key_iv: "af1c39f7585f62108ac32dddefb38d39"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.996454 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a31a6940de7485d88921050d690261f"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "7178a95e5643b1531425f420073879b2"
server_key_iv: "af1c39f7585f62108ac32dddefb38d39"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.000331 20084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:06.002828 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.004279 20084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:06.004415 20084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4a31a6940de7485d88921050d690261f"
format_stamp: "Formatted at 2026-09-29 18:35:05 on dist-test-slave-fmv8"
server_key: "7178a95e5643b1531425f420073879b2"
server_key_iv: "af1c39f7585f62108ac32dddefb38d39"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.004528 20084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.027200 20084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.027959 20084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.028214 20084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.035360 20084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37201
I20260929 18:35:06.035378 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37201 every 8 connection(s)
I20260929 18:35:06.036373 20084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:06.039248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20084
I20260929 18:35:06.039422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:06.039688 20151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.039722 392 external_mini_cluster.cc:1428] Setting key 5b5283747c699b793e0fde0a2d125398
I20260929 18:35:06.045831 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: Bootstrap starting.
I20260929 18:35:06.048034 20151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.048839 20151 log.cc:824] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.049187 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.041187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.98.62:37201)
0929 18:35:06.041876 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.041897 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.041944 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:35:06.042667 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.042684 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.043088 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.043425 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.043864 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.044912 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.045794 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.046309 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.046469 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.047405 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.047411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.047423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:06.047819 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.047874 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:06.047894 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.048004 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.048121 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.048434 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260929 18:35:06.051005 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: No bootstrap required, opened a new log
I20260929 18:35:06.054554 20151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } }
I20260929 18:35:06.054767 20151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.054800 20151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a31a6940de7485d88921050d690261f, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.055140 20151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } }
I20260929 18:35:06.055245 20151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:06.055281 20151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:06.055349 20151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.056227 20151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } }
I20260929 18:35:06.056512 20151 leader_election.cc:302] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a31a6940de7485d88921050d690261f; no voters:
I20260929 18:35:06.056830 20151 leader_election.cc:288] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:06.056973 20156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:06.057227 20156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [term 1 LEADER]: Becoming Leader. State: Replica: 4a31a6940de7485d88921050d690261f, State: Running, Role: LEADER
I20260929 18:35:06.057518 20156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } }
I20260929 18:35:06.058053 20151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:06.059360 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a31a6940de7485d88921050d690261f. Latest consensus state: current_term: 1 leader_uuid: "4a31a6940de7485d88921050d690261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } } }
I20260929 18:35:06.059471 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a31a6940de7485d88921050d690261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a31a6940de7485d88921050d690261f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37201 } } }
I20260929 18:35:06.059598 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:06.059479 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:06.060456 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:06.063035 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:06.068554 20165 catalog_manager.cc:1347] Generated new cluster ID: 549806cf93be481bb25f9f4865f18912
I20260929 18:35:06.068638 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:06.075778 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:06.076784 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:06.081195 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: Generated new TSK 0
I20260929 18:35:06.081868 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:06.088903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--builtin_ntp_servers=127.0.98.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:35:06.202123 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.202401 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.202472 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.206664 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.206725 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.206820 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 18:35:06.211768 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:06.212975 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.214406 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.224236 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.224234 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.224397 20175 server_base.cc:1034] running on GCE node
W20260929 18:35:06.224253 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.225099 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.225739 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.226912 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790706906226877 us; error 47 us; skew 500 ppm
I20260929 18:35:06.228878 20175 webserver.cc:460] Webserver started at http://127.0.98.1:36897/ using document root <none> and password file <none>
I20260929 18:35:06.229432 20175 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.229491 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.229662 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:06.231755 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7d64a6403e04495a3a0da42d71ba5ff"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "2145553830e33bc25215d425116644cc"
server_key_iv: "55952abc5fbb87e6f2ddc07b21ff285e"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.232447 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7d64a6403e04495a3a0da42d71ba5ff"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "2145553830e33bc25215d425116644cc"
server_key_iv: "55952abc5fbb87e6f2ddc07b21ff285e"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.236351 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:35:06.238860 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.240314 20175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:06.240453 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal
uuid: "f7d64a6403e04495a3a0da42d71ba5ff"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "2145553830e33bc25215d425116644cc"
server_key_iv: "55952abc5fbb87e6f2ddc07b21ff285e"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.240572 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.262611 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.263360 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.263561 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.264209 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:06.265221 20175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:06.265292 20175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.265362 20175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:06.265406 20175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.274945 20175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33647
I20260929 18:35:06.274961 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33647 every 8 connection(s)
I20260929 18:35:06.275897 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:06.285347 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.277693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44715 (local address 127.0.98.62:37201)
0929 18:35:06.277851 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.277854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.279046 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 18:35:06.280500 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.280504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.280565 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.280642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.281557 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.282098 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.283278 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.283507 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.283618 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.284800 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.284803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.284806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:06.284821 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.284848 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:06.284857 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.284929 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.285097 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.285163 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:35:06.285523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20175
I20260929 18:35:06.285650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:06.285953 392 external_mini_cluster.cc:1428] Setting key 0b6f7f121ac911e8783ffe0f3b4c6ee6
I20260929 18:35:06.286311 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.278105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37201 (local address 127.0.98.1:44715)
0929 18:35:06.278785 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.278831 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.280280 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.280843 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.280850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.280935 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:06.281411 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.281423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.282255 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.282257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.283151 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.283160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.283665 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.283670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.283840 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.284531 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:06.284533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:06.284574 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:06.284638 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:06.284950 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:06.284964 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:06.284968 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.285085 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.285377 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260929 18:35:06.287695 20303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37201
I20260929 18:35:06.287930 20303 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:06.288095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--builtin_ntp_servers=127.0.98.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:35:06.288524 20303 heartbeater.cc:511] Master 127.0.98.62:37201 requested a full tablet report, sending...
I20260929 18:35:06.290023 20115 ts_manager.cc:194] Registered new tserver with Master: f7d64a6403e04495a3a0da42d71ba5ff (127.0.98.1:33647)
I20260929 18:35:06.291165 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44715
W20260929 18:35:06.401697 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.401978 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.402017 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.406170 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.406231 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.406288 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 18:35:06.411242 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:06.412321 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.413681 20306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.423578 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.423600 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.423740 20306 server_base.cc:1034] running on GCE node
W20260929 18:35:06.423583 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.424383 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.425009 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.426196 20306 hybrid_clock.cc:648] HybridClock initialized: now 1790706906426164 us; error 46 us; skew 500 ppm
I20260929 18:35:06.431598 20306 webserver.cc:460] Webserver started at http://127.0.98.2:33079/ using document root <none> and password file <none>
I20260929 18:35:06.432260 20306 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.432351 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.432564 20306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:06.434706 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data/instance:
uuid: "28a83852be9947f48936415634ee875e"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "4d95921fd7273afb9772132e1954a0e9"
server_key_iv: "47233cf48ef53063378838cf770693c5"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.435295 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal/instance:
uuid: "28a83852be9947f48936415634ee875e"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "4d95921fd7273afb9772132e1954a0e9"
server_key_iv: "47233cf48ef53063378838cf770693c5"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.439373 20306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:06.442039 20320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.443300 20306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:06.443446 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal
uuid: "28a83852be9947f48936415634ee875e"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "4d95921fd7273afb9772132e1954a0e9"
server_key_iv: "47233cf48ef53063378838cf770693c5"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.443601 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.467549 20306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.468353 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.468559 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.469185 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:06.470152 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:06.470222 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.470287 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:06.470327 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.479650 20306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43757
I20260929 18:35:06.479687 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43757 every 8 connection(s)
I20260929 18:35:06.480624 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:06.486184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20306
I20260929 18:35:06.486318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:06.486644 392 external_mini_cluster.cc:1428] Setting key 67bfb835fd0d10d1bd583904337e8ac3
I20260929 18:35:06.488821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--builtin_ntp_servers=127.0.98.20:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:35:06.498369 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.482543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34285 (local address 127.0.98.62:37201)
0929 18:35:06.482687 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.482690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.483795 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 18:35:06.485245 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.485248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.485307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.485409 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.486520 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.487176 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.488686 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.488971 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.489071 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.497815 (+ 8744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.497819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.497821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:06.497838 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.497860 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:06.497869 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.497935 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.498158 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.498228 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:35:06.499568 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.482968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37201 (local address 127.0.98.2:34285)
0929 18:35:06.483640 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.483680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.485030 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.485578 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.485585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.485670 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:06.486345 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.486375 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.487319 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.487322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.488568 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.488582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.489152 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.489159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.489402 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.497388 (+ 7986us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:06.497393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:06.497469 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:06.497592 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:06.497966 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:06.497981 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:06.497991 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.498147 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.498536 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260929 18:35:06.500816 20434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37201
I20260929 18:35:06.501066 20434 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:06.501605 20434 heartbeater.cc:511] Master 127.0.98.62:37201 requested a full tablet report, sending...
I20260929 18:35:06.502679 20115 ts_manager.cc:194] Registered new tserver with Master: 28a83852be9947f48936415634ee875e (127.0.98.2:43757)
I20260929 18:35:06.503257 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34285
W20260929 18:35:06.623075 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.623339 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.623414 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.627431 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.627507 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.627589 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 18:35:06.632867 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:06.633948 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.635347 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.644797 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.644699 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.644819 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.645051 20437 server_base.cc:1034] running on GCE node
I20260929 18:35:06.645547 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.646189 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.647359 20437 hybrid_clock.cc:648] HybridClock initialized: now 1790706906647318 us; error 55 us; skew 500 ppm
I20260929 18:35:06.649155 20437 webserver.cc:460] Webserver started at http://127.0.98.3:41601/ using document root <none> and password file <none>
I20260929 18:35:06.649725 20437 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.649813 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.650013 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:06.652033 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data/instance:
uuid: "de8e4673aa33459f849eec80e96094e3"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "68f121aceb5e7b418f70fbc3bc470a03"
server_key_iv: "ce3ca165231d9c53eb9d6f771236b270"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.652637 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de8e4673aa33459f849eec80e96094e3"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "68f121aceb5e7b418f70fbc3bc470a03"
server_key_iv: "ce3ca165231d9c53eb9d6f771236b270"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.656453 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:06.658897 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.660321 20437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:06.660444 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal
uuid: "de8e4673aa33459f849eec80e96094e3"
format_stamp: "Formatted at 2026-09-29 18:35:06 on dist-test-slave-fmv8"
server_key: "68f121aceb5e7b418f70fbc3bc470a03"
server_key_iv: "ce3ca165231d9c53eb9d6f771236b270"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.660557 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.683614 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.684394 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.684602 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.685221 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:06.686203 20437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:06.686269 20437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.686336 20437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:06.686375 20437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.695369 20437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37485
I20260929 18:35:06.695394 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37485 every 8 connection(s)
I20260929 18:35:06.696468 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:06.705225 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.698302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53823 (local address 127.0.98.62:37201)
0929 18:35:06.698442 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.698445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.699485 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 18:35:06.700919 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.700921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.700969 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.701043 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.701892 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.702404 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.703522 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.703738 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.703840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.704839 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.704842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.704844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:06.704855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.704872 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:06.704880 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.704933 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.705061 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.705112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:35:06.705874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20437
I20260929 18:35:06.705967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:06.706153 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.698692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37201 (local address 127.0.98.3:53823)
0929 18:35:06.699356 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.699397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.700729 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.701198 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.701205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.701294 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:06.701778 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.701791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.702504 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.702506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.703416 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.703425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.703878 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.703881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.704019 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.704586 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:35:06.704588 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:06.704626 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:35:06.704685 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:06.704945 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:06.704951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:06.704955 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.705046 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.705290 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260929 18:35:06.706269 392 external_mini_cluster.cc:1428] Setting key 42db0b86c174516ba55ad1e9966d2029
I20260929 18:35:06.707470 20565 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37201
I20260929 18:35:06.707698 20565 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:06.708312 20565 heartbeater.cc:511] Master 127.0.98.62:37201 requested a full tablet report, sending...
I20260929 18:35:06.709205 20114 ts_manager.cc:194] Registered new tserver with Master: de8e4673aa33459f849eec80e96094e3 (127.0.98.3:37485)
I20260929 18:35:06.709754 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53823
I20260929 18:35:06.720119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:06.726040 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.722375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.98.62:37201)
0929 18:35:06.722556 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.722563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.722774 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:35:06.722846 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.722848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.722892 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.722962 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.723205 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.723728 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.724627 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.724844 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.724927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.725700 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.725703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.725706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:06.725717 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.725735 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:06.725743 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.725806 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.725881 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.725931 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:35:06.728850 20114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:06.732053 20114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:06.746795 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.741224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.0.98.2:43757)
0929 18:35:06.741844 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.741858 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.741882 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:35:06.741958 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.741966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.742147 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.742322 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.742717 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.743524 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.745313 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.745969 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.746087 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.746286 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.746458 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.746627 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":86,"threads_started":1}
I20260929 18:35:06.746788 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.740429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33647 (local address 127.0.0.1:37542)
0929 18:35:06.741359 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.741411 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.741541 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.742306 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.742311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.742345 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.742608 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.742615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.744054 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.744057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.745131 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.745140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.746214 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.746217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.746335 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.746357 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.746408 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.746553 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":104,"spinlock_wait_cycles":9856,"thread_start_us":68,"threads_started":1}
I20260929 18:35:06.746795 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.740827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43757 (local address 127.0.0.1:38886)
0929 18:35:06.741455 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.741470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.741563 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.742384 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.742386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.742403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.742609 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.742614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.743625 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.743631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.745115 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.745126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.746086 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.746090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.746216 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.746305 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.746427 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.746561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":174,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260929 18:35:06.747440 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.741067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.0.98.3:37485)
0929 18:35:06.741822 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.741833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.741855 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:35:06.741936 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.741940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.742162 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.742661 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.742826 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.743938 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.745401 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.746143 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.746241 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.746429 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.746940 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.747042 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":112,"threads_started":1}
I20260929 18:35:06.747663 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.740578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.0.98.1:33647)
0929 18:35:06.741821 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.741830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.741855 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:35:06.741931 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.741936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.742138 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.742314 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.742749 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.743917 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.745278 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.746068 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.747166 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.747355 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.747403 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.747503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":104,"threads_started":1}
I20260929 18:35:06.746867 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.740600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37485 (local address 127.0.0.1:42624)
0929 18:35:06.741359 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.741411 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.741541 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.742310 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.742314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.742346 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.742505 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.742512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.744040 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.744043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.745292 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.745302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.746485 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.746489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.746568 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.746588 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.746658 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.746718 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":68,"spinlock_wait_cycles":12672,"thread_start_us":136,"threads_started":1}
I20260929 18:35:06.750075 20237 tablet_service.cc:1467] Processing CreateTablet for tablet e6d51917929d4ca3b066797ef9e9baae (DEFAULT_TABLE table=test-table [id=a2932193950c44c4938e6e2e31bb82f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.750192 20499 tablet_service.cc:1467] Processing CreateTablet for tablet e6d51917929d4ca3b066797ef9e9baae (DEFAULT_TABLE table=test-table [id=a2932193950c44c4938e6e2e31bb82f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.750192 20368 tablet_service.cc:1467] Processing CreateTablet for tablet e6d51917929d4ca3b066797ef9e9baae (DEFAULT_TABLE table=test-table [id=a2932193950c44c4938e6e2e31bb82f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.751055 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d51917929d4ca3b066797ef9e9baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.751171 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d51917929d4ca3b066797ef9e9baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.751171 20368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d51917929d4ca3b066797ef9e9baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.757102 20582 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Bootstrap starting.
I20260929 18:35:06.758355 20583 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Bootstrap starting.
I20260929 18:35:06.759858 20582 tablet_bootstrap.cc:654] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.760440 20584 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Bootstrap starting.
I20260929 18:35:06.760469 20583 tablet_bootstrap.cc:654] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.760773 20582 log.cc:824] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.761260 20583 log.cc:824] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.762441 20584 tablet_bootstrap.cc:654] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.763100 20582 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: No bootstrap required, opened a new log
I20260929 18:35:06.763187 20583 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: No bootstrap required, opened a new log
I20260929 18:35:06.763239 20584 log.cc:824] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.763304 20582 ts_tablet_manager.cc:1397] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:35:06.763381 20583 ts_tablet_manager.cc:1397] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:35:06.765226 20584 tablet_bootstrap.cc:492] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: No bootstrap required, opened a new log
I20260929 18:35:06.765400 20584 ts_tablet_manager.cc:1397] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:35:06.766883 20582 raft_consensus.cc:348] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.767133 20582 raft_consensus.cc:374] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.767184 20582 raft_consensus.cc:729] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d64a6403e04495a3a0da42d71ba5ff, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.767653 20582 consensus_queue.cc:260] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.767702 20583 raft_consensus.cc:348] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.767963 20583 raft_consensus.cc:374] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.768014 20583 raft_consensus.cc:729] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a83852be9947f48936415634ee875e, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.768499 20583 consensus_queue.cc:260] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.768741 20582 ts_tablet_manager.cc:1428] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:35:06.769322 20303 heartbeater.cc:503] Master 127.0.98.62:37201 was elected leader, sending a full tablet report...
I20260929 18:35:06.769411 20583 ts_tablet_manager.cc:1428] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:35:06.769542 20584 raft_consensus.cc:348] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.769747 20434 heartbeater.cc:503] Master 127.0.98.62:37201 was elected leader, sending a full tablet report...
I20260929 18:35:06.769786 20584 raft_consensus.cc:374] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.769840 20584 raft_consensus.cc:729] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e4673aa33459f849eec80e96094e3, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.770198 20584 consensus_queue.cc:260] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.770896 20565 heartbeater.cc:503] Master 127.0.98.62:37201 was elected leader, sending a full tablet report...
I20260929 18:35:06.771271 20584 ts_tablet_manager.cc:1428] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 18:35:06.777011 20304 tablet.cc:2369] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:06.947474 20566 tablet.cc:2369] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:06.975252 20590 raft_consensus.cc:482] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:06.975469 20590 raft_consensus.cc:504] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.976989 20590 leader_election.cc:288] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d64a6403e04495a3a0da42d71ba5ff (127.0.98.1:33647), 28a83852be9947f48936415634ee875e (127.0.98.2:43757)
I20260929 18:35:06.981076 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.977195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33647 (local address 127.0.98.3:47783)
0929 18:35:06.977470 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.977487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.977612 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.977904 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.977907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.977927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.978062 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.978072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.978908 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.978912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.979798 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.979807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.980596 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.980600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.980715 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.980775 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.980852 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.980913 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":19}
I20260929 18:35:06.981140 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.977303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47783 (local address 127.0.98.1:33647)
0929 18:35:06.977448 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.977455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.977561 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:35:06.977719 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.977722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.977784 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.977865 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.978215 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.978793 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.979921 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.980476 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.980560 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.980612 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.980900 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.980990 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:35:06.981215 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.977399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36581 (local address 127.0.98.2:43757)
0929 18:35:06.977522 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.977525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.977879 (+ 354us) server_negotiation.cc:407] Connection header received
0929 18:35:06.977942 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.977944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.977993 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.978077 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.978436 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.978996 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.980100 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.980585 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.980699 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.980765 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.981004 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.981088 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:06.981725 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.977303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43757 (local address 127.0.98.3:36581)
0929 18:35:06.977749 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.977762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.977870 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.978112 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.978114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.978132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.978284 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.978290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.979104 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.979106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.979949 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.979958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.980723 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.980726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.980859 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.980883 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.980960 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.981022 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":19,"thread_start_us":140,"threads_started":1}
W20260929 18:35:06.981793 20435 tablet.cc:2369] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:06.981889 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d51917929d4ca3b066797ef9e9baae" candidate_uuid: "de8e4673aa33459f849eec80e96094e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" is_pre_election: true
I20260929 18:35:06.982210 20257 raft_consensus.cc:2454] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e4673aa33459f849eec80e96094e3 in term 0.
I20260929 18:35:06.982399 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d51917929d4ca3b066797ef9e9baae" candidate_uuid: "de8e4673aa33459f849eec80e96094e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83852be9947f48936415634ee875e" is_pre_election: true
I20260929 18:35:06.982676 20388 raft_consensus.cc:2454] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e4673aa33459f849eec80e96094e3 in term 0.
I20260929 18:35:06.982748 20454 leader_election.cc:302] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e4673aa33459f849eec80e96094e3, f7d64a6403e04495a3a0da42d71ba5ff; no voters:
I20260929 18:35:06.983069 20590 raft_consensus.cc:2781] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:06.983150 20590 raft_consensus.cc:482] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:06.983199 20590 raft_consensus.cc:3037] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.984248 20590 raft_consensus.cc:504] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.984637 20590 leader_election.cc:288] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [CANDIDATE]: Term 1 election: Requested vote from peers f7d64a6403e04495a3a0da42d71ba5ff (127.0.98.1:33647), 28a83852be9947f48936415634ee875e (127.0.98.2:43757)
I20260929 18:35:06.984915 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d51917929d4ca3b066797ef9e9baae" candidate_uuid: "de8e4673aa33459f849eec80e96094e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d64a6403e04495a3a0da42d71ba5ff"
I20260929 18:35:06.984987 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d51917929d4ca3b066797ef9e9baae" candidate_uuid: "de8e4673aa33459f849eec80e96094e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a83852be9947f48936415634ee875e"
I20260929 18:35:06.985034 20257 raft_consensus.cc:3037] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.985143 20388 raft_consensus.cc:3037] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.986090 20257 raft_consensus.cc:2454] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e4673aa33459f849eec80e96094e3 in term 1.
I20260929 18:35:06.986205 20388 raft_consensus.cc:2454] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e4673aa33459f849eec80e96094e3 in term 1.
I20260929 18:35:06.986438 20454 leader_election.cc:302] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e4673aa33459f849eec80e96094e3, f7d64a6403e04495a3a0da42d71ba5ff; no voters:
I20260929 18:35:06.986608 20590 raft_consensus.cc:2781] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:06.986841 20590 raft_consensus.cc:686] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 1 LEADER]: Becoming Leader. State: Replica: de8e4673aa33459f849eec80e96094e3, State: Running, Role: LEADER
I20260929 18:35:06.987169 20590 consensus_queue.cc:237] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } }
I20260929 18:35:06.990034 20115 catalog_manager.cc:5495] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to de8e4673aa33459f849eec80e96094e3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "de8e4673aa33459f849eec80e96094e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e4673aa33459f849eec80e96094e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:07.039357 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.035907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.98.3:37485)
0929 18:35:07.036053 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:07.036059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:07.036076 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:07.036212 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:07.036214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:07.036268 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:07.036377 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:07.036599 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:07.037132 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:07.037969 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:07.038196 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:07.038298 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:07.038593 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:07.038976 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:07.039117 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:07.039152 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 18:35:07.039201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:07.045545 20257 raft_consensus.cc:1261] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff [term 1 FOLLOWER]: Refusing update from remote peer de8e4673aa33459f849eec80e96094e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:07.045545 20388 raft_consensus.cc:1261] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 1 FOLLOWER]: Refusing update from remote peer de8e4673aa33459f849eec80e96094e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:07.046298 20590 consensus_queue.cc:1035] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a83852be9947f48936415634ee875e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:07.046500 20592 consensus_queue.cc:1035] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d64a6403e04495a3a0da42d71ba5ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:07.063112 20115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:07.063289 20115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:07.066058 20115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a31a6940de7485d88921050d690261f: Sending DeleteTablet for 3 replicas of tablet e6d51917929d4ca3b066797ef9e9baae
I20260929 18:35:07.067005 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d51917929d4ca3b066797ef9e9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:07 UTC) from {username='slave'} at 127.0.0.1:37542
I20260929 18:35:07.067004 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d51917929d4ca3b066797ef9e9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:07 UTC) from {username='slave'} at 127.0.0.1:42624
I20260929 18:35:07.067035 20368 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d51917929d4ca3b066797ef9e9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:07 UTC) from {username='slave'} at 127.0.0.1:38886
I20260929 18:35:07.067579 20605 tablet_replica.cc:330] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: stopping tablet replica
I20260929 18:35:07.067941 20605 raft_consensus.cc:2229] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:07.068089 20607 tablet_replica.cc:330] T e6d51917929d4ca3b066797ef9e9baae P f7d64a6403e04495a3a0da42d71ba5ff: stopping tablet replica
I20260929 18:35:07.068174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20175
I20260929 18:35:07.068358 20605 raft_consensus.cc:2258] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:07.068768 20606 tablet_replica.cc:330] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: stopping tablet replica
I20260929 18:35:07.069098 20606 raft_consensus.cc:2229] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:07.069535 20606 raft_consensus.cc:2258] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:07.070859 20605 ts_tablet_manager.cc:1905] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:07.071002 20606 ts_tablet_manager.cc:1905] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:07.073885 20605 ts_tablet_manager.cc:1918] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:07.073983 20605 log.cc:1196] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-1/wal/wals/e6d51917929d4ca3b066797ef9e9baae
I20260929 18:35:07.074329 20605 ts_tablet_manager.cc:1939] T e6d51917929d4ca3b066797ef9e9baae P 28a83852be9947f48936415634ee875e: Deleting consensus metadata
I20260929 18:35:07.074491 20606 ts_tablet_manager.cc:1918] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:07.074584 20606 log.cc:1196] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Deleting WAL directory at /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706655775522-392-0/minicluster-data/ts-2/wal/wals/e6d51917929d4ca3b066797ef9e9baae
I20260929 18:35:07.074884 20606 ts_tablet_manager.cc:1939] T e6d51917929d4ca3b066797ef9e9baae P de8e4673aa33459f849eec80e96094e3: Deleting consensus metadata
I20260929 18:35:07.075785 20102 catalog_manager.cc:4841] TS 28a83852be9947f48936415634ee875e (127.0.98.2:43757): tablet e6d51917929d4ca3b066797ef9e9baae (table test-table [id=a2932193950c44c4938e6e2e31bb82f0]) successfully deleted
I20260929 18:35:07.076555 20099 catalog_manager.cc:4841] TS de8e4673aa33459f849eec80e96094e3 (127.0.98.3:37485): tablet e6d51917929d4ca3b066797ef9e9baae (table test-table [id=a2932193950c44c4938e6e2e31bb82f0]) successfully deleted
W20260929 18:35:07.076930 20101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33647 (error 108)
I20260929 18:35:07.077168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20306
I20260929 18:35:07.082584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20437
I20260929 18:35:07.082927 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.082591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33647 (local address 127.0.0.1:37554)
0929 18:35:07.082734 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.082820 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:35:07.084264 20101 catalog_manager.cc:4584] TS f7d64a6403e04495a3a0da42d71ba5ff (127.0.98.1:33647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6d51917929d4ca3b066797ef9e9baae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33647: connect: Connection refused (error 111)
I20260929 18:35:07.088104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20084
2026-09-29T18:35:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1264 ms)
[----------] 6 tests from EncryptionPolicyTest (10834 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:07 dist-test-slave-fmv8 krb5kdc[20611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:07 dist-test-slave-fmv8 krb5kdc[20611](info): set up 2 sockets
Sep 29 18:35:07 dist-test-slave-fmv8 krb5kdc[20611](info): commencing operation
krb5kdc: starting...
W20260929 18:35:08.120715 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:08.243278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:08.355037 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:08.355291 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:08.355334 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:08.359366 20624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:08.359424 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:08.359444 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:08.359462 20624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:08.359477 20624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:08.364648 20624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:08.365670 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:08.366626 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:35:08.375129 20624 server_base.cc:1034] running on GCE node
W20260929 18:35:08.375047 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:08.375053 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:08.375047 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:08.375916 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:08.377064 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:08.378239 20624 hybrid_clock.cc:648] HybridClock initialized: now 1790706908378208 us; error 44 us; skew 500 ppm
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:08.381127 20624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:08.382053 20624 webserver.cc:460] Webserver started at http://127.0.0.1:36517/ using document root <none> and password file <none>
I20260929 18:35:08.382619 20624 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:08.382709 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:08.382936 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:08.385033 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "00fec91ec1b447829413fd661636704e"
format_stamp: "Formatted at 2026-09-29 18:35:08 on dist-test-slave-fmv8"
server_key: "f86cd1d50aa0e10cb3b48b7b20fc8932"
server_key_iv: "c6a2acd476720e1e9c137572a756de95"
server_key_version: "encryptionkey@0"
I20260929 18:35:08.385587 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "00fec91ec1b447829413fd661636704e"
format_stamp: "Formatted at 2026-09-29 18:35:08 on dist-test-slave-fmv8"
server_key: "f86cd1d50aa0e10cb3b48b7b20fc8932"
server_key_iv: "c6a2acd476720e1e9c137572a756de95"
server_key_version: "encryptionkey@0"
I20260929 18:35:08.389276 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:08.391695 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:08.393028 20624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:08.393170 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "00fec91ec1b447829413fd661636704e"
format_stamp: "Formatted at 2026-09-29 18:35:08 on dist-test-slave-fmv8"
server_key: "f86cd1d50aa0e10cb3b48b7b20fc8932"
server_key_iv: "c6a2acd476720e1e9c137572a756de95"
server_key_version: "encryptionkey@0"
I20260929 18:35:08.393286 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:08.419492 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:08.422984 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:08.423198 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:08.430514 20624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46039
I20260929 18:35:08.430531 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46039 every 8 connection(s)
I20260929 18:35:08.431469 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:08.434214 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:08.439595 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e: Bootstrap starting.
I20260929 18:35:08.439685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20624
I20260929 18:35:08.439867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:08.440223 392 external_mini_cluster.cc:1428] Setting key d246fbff208acb26999ea1510ad6a318
I20260929 18:35:08.442406 20692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:08.443279 20692 log.cc:824] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:08.445462 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e: No bootstrap required, opened a new log
I20260929 18:35:08.449456 20692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } }
I20260929 18:35:08.449733 20692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:08.449783 20692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fec91ec1b447829413fd661636704e, State: Initialized, Role: FOLLOWER
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:35:08.450217 20692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } }
I20260929 18:35:08.450335 20692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:08.450431 20692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:08.450557 20692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:08.451467 20692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } }
I20260929 18:35:08.451798 20692 leader_election.cc:302] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00fec91ec1b447829413fd661636704e; no voters:
I20260929 18:35:08.452105 20692 leader_election.cc:288] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:08.452241 20697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:08.452471 20697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [term 1 LEADER]: Becoming Leader. State: Replica: 00fec91ec1b447829413fd661636704e, State: Running, Role: LEADER
I20260929 18:35:08.452814 20697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } }
I20260929 18:35:08.453394 20692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:08.454555 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00fec91ec1b447829413fd661636704e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } } }
I20260929 18:35:08.454691 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:08.455050 20706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:08.455596 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00fec91ec1b447829413fd661636704e. Latest consensus state: current_term: 1 leader_uuid: "00fec91ec1b447829413fd661636704e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00fec91ec1b447829413fd661636704e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46039 } } }
I20260929 18:35:08.455675 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:08.456559 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.441641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43285 (local address 127.0.0.1:46039)
0929 18:35:08.442066 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:08.442085 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:08.442127 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:35:08.442984 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:08.443006 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:08.443437 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:08.443795 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:08.444630 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.446029 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.446888 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.447597 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.447749 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.450585 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:08.450592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:08.450610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:08.450643 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:08.452740 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.453389 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.453395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.453400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.453531 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.453986 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.453989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.453991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.454521 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:08.454750 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:08.455239 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 18:35:08.455478 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260929 18:35:08.458266 20706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:08.459560 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:08.464663 20706 catalog_manager.cc:1347] Generated new cluster ID: c86dbffde2e441a59d8fc24484b99461
I20260929 18:35:08.464725 20706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:08.470213 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.461692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34539 (local address 127.0.0.1:46039)
0929 18:35:08.461789 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:08.461793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:08.462065 (+ 272us) server_negotiation.cc:407] Connection header received
0929 18:35:08.462170 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:08.462173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:08.462218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:08.462299 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:08.463151 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.463662 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.464646 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.464978 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.465052 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.466795 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:08.466799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:08.466801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:08.466824 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:08.468431 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.468926 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.468929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.468931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.469004 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.469306 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.469309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.469311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.469487 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:08.469621 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:08.469976 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 18:35:08.470067 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:35:08.487221 20706 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:08.488269 20706 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:08.495611 20706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00fec91ec1b447829413fd661636704e: Generated new TSK 0
I20260929 18:35:08.496405 20706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:08.501706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20624
W20260929 18:35:08.507285 20609 connection.cc:535] client connection to 127.0.0.1:46039 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46039): Connection reset by peer (error 104)
2026-09-29T18:35:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20723](info): set up 2 sockets
Sep 29 18:35:08 dist-test-slave-fmv8 krb5kdc[20723](info): commencing operation
krb5kdc: starting...
W20260929 18:35:09.540452 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:09 dist-test-slave-fmv8 krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:09.659834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:09.768433 20736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:09.768699 20736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:09.768743 20736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:09.772733 20736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:09.772796 20736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:09.772815 20736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:09.772832 20736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:09.772847 20736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:09.778046 20736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:09.779030 20736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:09.779982 20736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:09.788784 20742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:09.788920 20744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:09.788848 20741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:09.789042 20736 server_base.cc:1034] running on GCE node
I20260929 18:35:09.789711 20736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:09.790982 20736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:09.792171 20736 hybrid_clock.cc:648] HybridClock initialized: now 1790706909792125 us; error 57 us; skew 500 ppm
Sep 29 18:35:09 dist-test-slave-fmv8 krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:09.795439 20736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:09.796365 20736 webserver.cc:460] Webserver started at http://127.0.0.1:42967/ using document root <none> and password file <none>
I20260929 18:35:09.796944 20736 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:09.797034 20736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:09.797262 20736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:09.799332 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "4c89296f820647c49f2a83d0274f5533"
format_stamp: "Formatted at 2026-09-29 18:35:09 on dist-test-slave-fmv8"
server_key: "18b4d1db387f4d856ce8b90134e5f318"
server_key_iv: "2c45745f53c67f5e5da1225ae615bbbc"
server_key_version: "encryptionkey@0"
I20260929 18:35:09.799873 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c89296f820647c49f2a83d0274f5533"
format_stamp: "Formatted at 2026-09-29 18:35:09 on dist-test-slave-fmv8"
server_key: "18b4d1db387f4d856ce8b90134e5f318"
server_key_iv: "2c45745f53c67f5e5da1225ae615bbbc"
server_key_version: "encryptionkey@0"
I20260929 18:35:09.803440 20736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:35:09.805806 20751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:09.806948 20736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:09.807080 20736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4c89296f820647c49f2a83d0274f5533"
format_stamp: "Formatted at 2026-09-29 18:35:09 on dist-test-slave-fmv8"
server_key: "18b4d1db387f4d856ce8b90134e5f318"
server_key_iv: "2c45745f53c67f5e5da1225ae615bbbc"
server_key_version: "encryptionkey@0"
I20260929 18:35:09.807188 20736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:09.830819 20736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:09.833963 20736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:09.834179 20736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:09.841137 20736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34253
I20260929 18:35:09.841157 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34253 every 8 connection(s)
I20260929 18:35:09.842057 20736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:09.844769 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:09.845736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20736
I20260929 18:35:09.845884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:09.846225 392 external_mini_cluster.cc:1428] Setting key 329efbf1125567af46c2932b1ecfd932
I20260929 18:35:09.851162 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533: Bootstrap starting.
I20260929 18:35:09.853777 20804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:09.854616 20804 log.cc:824] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:09.855319 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:09.847615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34717 (local address 127.0.0.1:34253)
0929 18:35:09.848073 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:09.848087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:09.848123 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:09.848791 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:09.848807 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:09.849137 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:09.849407 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:09.849764 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:09.850735 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:09.851885 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:09.852416 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:09.852537 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:09.853398 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:09.853403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:09.853415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:09.853818 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:09.853918 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:09.853942 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:09.854043 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:09.854244 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:35:09.854399 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260929 18:35:09.856863 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533: No bootstrap required, opened a new log
I20260929 18:35:09.858268 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:09.860823 20804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260929 18:35:09.861110 20804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:09.861193 20804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c89296f820647c49f2a83d0274f5533, State: Initialized, Role: FOLLOWER
I20260929 18:35:09.861595 20804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260929 18:35:09.861738 20804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:09.861822 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:09.861927 20804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:09.862808 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260929 18:35:09.863149 20804 leader_election.cc:302] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c89296f820647c49f2a83d0274f5533; no voters:
I20260929 18:35:09.863440 20804 leader_election.cc:288] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:09.863694 20814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:09.864194 20814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [term 1 LEADER]: Becoming Leader. State: Replica: 4c89296f820647c49f2a83d0274f5533, State: Running, Role: LEADER
I20260929 18:35:09.864396 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:09.860223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39939 (local address 127.0.0.1:34253)
0929 18:35:09.860349 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:09.860353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:09.860564 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:35:09.860672 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:09.860674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:09.860710 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:09.860772 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:09.861066 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:09.861758 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:09.862559 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:09.862812 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:09.862916 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:09.863860 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:09.863864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:09.863866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:09.863878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:09.863906 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:09.863921 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:09.864016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:09.864243 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 18:35:09.864280 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:35:09.864512 20814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260929 18:35:09.864799 20804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:09.866060 20815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c89296f820647c49f2a83d0274f5533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260929 18:35:09.866183 20815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:09.866475 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c89296f820647c49f2a83d0274f5533. Latest consensus state: current_term: 1 leader_uuid: "4c89296f820647c49f2a83d0274f5533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c89296f820647c49f2a83d0274f5533" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260929 18:35:09.866561 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:09.867254 20823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:09.869884 20823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:09.874904 20823 catalog_manager.cc:1347] Generated new cluster ID: cfc024d7a75d47e7b24f41d715c6f98b
I20260929 18:35:09.874974 20823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:09.900199 20823 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:09.901525 20823 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:09.911605 20823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c89296f820647c49f2a83d0274f5533: Generated new TSK 0
I20260929 18:35:09.912372 20823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:09.917887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20736
W20260929 18:35:09.923276 20721 connection.cc:535] client connection to 127.0.0.1:34253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34253): Connection reset by peer (error 104)
2026-09-29T18:35:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:09 dist-test-slave-fmv8 krb5kdc[20835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:09 dist-test-slave-fmv8 krb5kdc[20835](info): set up 2 sockets
Sep 29 18:35:09 dist-test-slave-fmv8 krb5kdc[20835](info): commencing operation
krb5kdc: starting...
W20260929 18:35:10.984511 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:11 dist-test-slave-fmv8 krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:11.116636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:11.228914 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:11.229259 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:11.229346 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:11.233554 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:11.233667 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:11.233703 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:11.233747 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:11.233778 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:11.239320 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:11.240536 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:11.241647 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:11.250329 20854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:11.250514 20848 server_base.cc:1034] running on GCE node
W20260929 18:35:11.250459 20853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:11.250310 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:11.251219 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:11.252589 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:11.253772 20848 hybrid_clock.cc:648] HybridClock initialized: now 1790706911253729 us; error 56 us; skew 500 ppm
Sep 29 18:35:11 dist-test-slave-fmv8 krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:11.257017 20848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:11.258033 20848 webserver.cc:460] Webserver started at http://127.0.0.1:42257/ using document root <none> and password file <none>
I20260929 18:35:11.258665 20848 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:11.258733 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:11.258955 20848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:11.261118 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "0dd884c90bb9476ba91c5b8188c2628f"
format_stamp: "Formatted at 2026-09-29 18:35:11 on dist-test-slave-fmv8"
server_key: "bb12fe9a27fed14c455e1065d8f79159"
server_key_iv: "ddd64e094f2d89ed6e9e463cfb5e2409"
server_key_version: "encryptionkey@0"
I20260929 18:35:11.261694 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dd884c90bb9476ba91c5b8188c2628f"
format_stamp: "Formatted at 2026-09-29 18:35:11 on dist-test-slave-fmv8"
server_key: "bb12fe9a27fed14c455e1065d8f79159"
server_key_iv: "ddd64e094f2d89ed6e9e463cfb5e2409"
server_key_version: "encryptionkey@0"
I20260929 18:35:11.265838 20848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:11.268889 20863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:11.270488 20848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:11.270659 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0dd884c90bb9476ba91c5b8188c2628f"
format_stamp: "Formatted at 2026-09-29 18:35:11 on dist-test-slave-fmv8"
server_key: "bb12fe9a27fed14c455e1065d8f79159"
server_key_iv: "ddd64e094f2d89ed6e9e463cfb5e2409"
server_key_version: "encryptionkey@0"
I20260929 18:35:11.270797 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:11.275667 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:11.278846 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:11.279063 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:11.287387 20848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35365
I20260929 18:35:11.287408 20915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35365 every 8 connection(s)
I20260929 18:35:11.288425 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:11.291958 20916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:11.293567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20848
I20260929 18:35:11.293715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:11.294032 392 external_mini_cluster.cc:1428] Setting key 9138d4b00dd4fb666f743a4ff2ddbb73
I20260929 18:35:11.299584 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f: Bootstrap starting.
I20260929 18:35:11.299878 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:11.295557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47761 (local address 127.0.0.1:35365)
0929 18:35:11.296088 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:11.296108 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:11.296179 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:35:11.296847 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:11.296865 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:11.297169 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:11.297535 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:11.298319 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:11.298328 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:11.298345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:11.298358 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:11.298424 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:11.298455 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:11.298615 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:11.298776 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:35:11.298955 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":140,"threads_started":1}
I20260929 18:35:11.302220 20916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:11.303088 20916 log.cc:824] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:11.303158 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:11.305260 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f: No bootstrap required, opened a new log
I20260929 18:35:11.307042 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:11.305286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36443 (local address 127.0.0.1:35365)
0929 18:35:11.305408 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:11.305413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:11.305595 (+ 182us) server_negotiation.cc:407] Connection header received
0929 18:35:11.305763 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:11.305766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:11.305800 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:11.305886 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:11.306638 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:11.306642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:11.306644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:11.306656 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:11.306685 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:11.306699 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:11.306798 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:11.306883 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 18:35:11.306924 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:11.309418 20916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } }
I20260929 18:35:11.309737 20916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:11.309788 20916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd884c90bb9476ba91c5b8188c2628f, State: Initialized, Role: FOLLOWER
I20260929 18:35:11.310241 20916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } }
I20260929 18:35:11.310365 20916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:11.310433 20916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:11.310560 20916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:11.311514 20916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } }
I20260929 18:35:11.311867 20916 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd884c90bb9476ba91c5b8188c2628f; no voters:
I20260929 18:35:11.312299 20916 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:11.312388 20926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:11.312624 20926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [term 1 LEADER]: Becoming Leader. State: Replica: 0dd884c90bb9476ba91c5b8188c2628f, State: Running, Role: LEADER
I20260929 18:35:11.312949 20926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } }
I20260929 18:35:11.313805 20916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:11.314922 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd884c90bb9476ba91c5b8188c2628f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } } }
I20260929 18:35:11.314919 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd884c90bb9476ba91c5b8188c2628f. Latest consensus state: current_term: 1 leader_uuid: "0dd884c90bb9476ba91c5b8188c2628f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd884c90bb9476ba91c5b8188c2628f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35365 } } }
I20260929 18:35:11.315093 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:11.315092 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:11.315939 20935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:11.319217 20935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:11.325824 20935 catalog_manager.cc:1347] Generated new cluster ID: 0fe2d5cbc2f3440e83fe4f3ebe743277
I20260929 18:35:11.325939 20935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:11.335186 20935 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:11.335587 20935 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:11.345299 20935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd884c90bb9476ba91c5b8188c2628f: Generated new TSK 0
I20260929 18:35:11.346122 20935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:11.351461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20848
2026-09-29T18:35:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:11 dist-test-slave-fmv8 krb5kdc[20947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:11 dist-test-slave-fmv8 krb5kdc[20947](info): set up 2 sockets
Sep 29 18:35:11 dist-test-slave-fmv8 krb5kdc[20947](info): commencing operation
krb5kdc: starting...
W20260929 18:35:12.400422 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:12.519960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:12.631857 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:12.632200 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:12.632313 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:12.636332 20960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:12.636405 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:12.636440 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:12.636477 20960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:12.636510 20960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:12.641760 20960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:12.642861 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:12.643900 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:12.652704 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:12.652814 20960 server_base.cc:1034] running on GCE node
W20260929 18:35:12.652829 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:12.652992 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:12.653800 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:12.654942 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:12.656139 20960 hybrid_clock.cc:648] HybridClock initialized: now 1790706912656116 us; error 34 us; skew 500 ppm
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:12.662642 20960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:12.663626 20960 webserver.cc:460] Webserver started at http://127.0.0.1:42705/ using document root <none> and password file <none>
I20260929 18:35:12.664258 20960 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:12.664353 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:12.664585 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:12.666622 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "d8c13349c068435c897a2ed567024e4f"
format_stamp: "Formatted at 2026-09-29 18:35:12 on dist-test-slave-fmv8"
server_key: "a168d4ae94c0aeab22c024506618b12f"
server_key_iv: "4329983f8be50b70d1c10c2dcad1e65b"
server_key_version: "encryptionkey@0"
I20260929 18:35:12.667199 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8c13349c068435c897a2ed567024e4f"
format_stamp: "Formatted at 2026-09-29 18:35:12 on dist-test-slave-fmv8"
server_key: "a168d4ae94c0aeab22c024506618b12f"
server_key_iv: "4329983f8be50b70d1c10c2dcad1e65b"
server_key_version: "encryptionkey@0"
I20260929 18:35:12.670881 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:35:12.673260 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:12.674475 20960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:12.674612 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "d8c13349c068435c897a2ed567024e4f"
format_stamp: "Formatted at 2026-09-29 18:35:12 on dist-test-slave-fmv8"
server_key: "a168d4ae94c0aeab22c024506618b12f"
server_key_iv: "4329983f8be50b70d1c10c2dcad1e65b"
server_key_version: "encryptionkey@0"
I20260929 18:35:12.674718 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:12.702914 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:12.706264 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:12.706477 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:12.713913 20960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44125
I20260929 18:35:12.713950 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44125 every 8 connection(s)
I20260929 18:35:12.714867 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:12.716096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 20960
I20260929 18:35:12.716308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:12.716604 392 external_mini_cluster.cc:1428] Setting key 8b42fe84beea848108ea0e7a4c329b05
I20260929 18:35:12.717850 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:12.723196 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f: Bootstrap starting.
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[20947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:35:12.725770 21028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:12.726663 21028 log.cc:824] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:12.729063 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f: No bootstrap required, opened a new log
I20260929 18:35:12.731216 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:12.717983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37355 (local address 127.0.0.1:44125)
0929 18:35:12.718400 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:12.718415 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:12.718450 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:35:12.719087 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:12.719103 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:12.719441 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:12.719774 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:12.720595 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:12.721583 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:12.722432 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:12.722907 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:12.723076 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:12.725971 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:12.725979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:12.725998 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:12.726037 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:12.727980 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:12.728511 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:12.728516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:12.728523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:12.728669 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:12.728969 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:12.728973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:12.728974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:12.729493 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:12.729678 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:12.729973 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:35:12.730260 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":138,"threads_started":1}
I20260929 18:35:12.733417 21028 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260929 18:35:12.733695 21028 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:12.733771 21028 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c13349c068435c897a2ed567024e4f, State: Initialized, Role: FOLLOWER
I20260929 18:35:12.734194 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:12.734236 21028 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260929 18:35:12.734431 21028 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:12.734499 21028 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:12.734596 21028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:12.735496 21028 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260929 18:35:12.735879 21028 leader_election.cc:302] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c13349c068435c897a2ed567024e4f; no voters:
I20260929 18:35:12.736231 21028 leader_election.cc:288] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:12.736300 21037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:12.736539 21037 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [term 1 LEADER]: Becoming Leader. State: Replica: d8c13349c068435c897a2ed567024e4f, State: Running, Role: LEADER
I20260929 18:35:12.736824 21037 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } }
I20260929 18:35:12.737505 21028 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:12.737943 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c13349c068435c897a2ed567024e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } } }
I20260929 18:35:12.738056 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:12.738255 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c13349c068435c897a2ed567024e4f. Latest consensus state: current_term: 1 leader_uuid: "d8c13349c068435c897a2ed567024e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c13349c068435c897a2ed567024e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44125 } } }
I20260929 18:35:12.738441 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:12.738456 21042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:12.741739 21042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:12.747617 21042 catalog_manager.cc:1347] Generated new cluster ID: 2f76268e7880410d93ec13a936d581f5
I20260929 18:35:12.747677 21042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:12.747997 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:12.736422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41561 (local address 127.0.0.1:44125)
0929 18:35:12.736533 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:12.736537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:12.738891 (+ 2354us) server_negotiation.cc:407] Connection header received
0929 18:35:12.739076 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:12.739079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:12.739134 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:12.739192 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:12.740216 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:12.741005 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:12.742130 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:12.742476 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:12.742554 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:12.744449 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:12.744452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:12.744455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:12.744481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:12.746164 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:12.746643 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:12.746646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:12.746648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:12.746740 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:12.747077 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:12.747081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:12.747082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:12.747299 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:12.747422 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:12.747761 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 18:35:12.747873 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:35:12.765559 21042 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:12.766551 21042 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:12.771164 21042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c13349c068435c897a2ed567024e4f: Generated new TSK 0
I20260929 18:35:12.771914 21042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:12.782964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 20960
2026-09-29T18:35:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[21059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[21059](info): set up 2 sockets
Sep 29 18:35:12 dist-test-slave-fmv8 krb5kdc[21059](info): commencing operation
krb5kdc: starting...
W20260929 18:35:13.848440 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:13 dist-test-slave-fmv8 krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:13.971243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:14.086994 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:14.087347 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:14.087432 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:14.091723 21072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:14.091851 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:14.091892 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:14.091952 21072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:14.092010 21072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:14.097630 21072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:14.098759 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:14.099810 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:14.108682 21080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:14.108654 21078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:14.108754 21072 server_base.cc:1034] running on GCE node
W20260929 18:35:14.108661 21077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:14.109462 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:14.110695 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:14.111899 21072 hybrid_clock.cc:648] HybridClock initialized: now 1790706914111869 us; error 46 us; skew 500 ppm
Sep 29 18:35:14 dist-test-slave-fmv8 krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:14.114890 21072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:14.115833 21072 webserver.cc:460] Webserver started at http://127.0.0.1:38183/ using document root <none> and password file <none>
I20260929 18:35:14.116465 21072 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:14.116555 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:14.116775 21072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:14.118816 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "5bc5a9a932a745c0a20b50e21ac28c5e"
format_stamp: "Formatted at 2026-09-29 18:35:14 on dist-test-slave-fmv8"
server_key: "c92102dd1ed9e6e6b355f43dddbbd092"
server_key_iv: "3b44432039f84198c84f0d84333dc06d"
server_key_version: "encryptionkey@0"
I20260929 18:35:14.119402 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bc5a9a932a745c0a20b50e21ac28c5e"
format_stamp: "Formatted at 2026-09-29 18:35:14 on dist-test-slave-fmv8"
server_key: "c92102dd1ed9e6e6b355f43dddbbd092"
server_key_iv: "3b44432039f84198c84f0d84333dc06d"
server_key_version: "encryptionkey@0"
I20260929 18:35:14.123307 21072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:14.125965 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:14.127251 21072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:14.127393 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "5bc5a9a932a745c0a20b50e21ac28c5e"
format_stamp: "Formatted at 2026-09-29 18:35:14 on dist-test-slave-fmv8"
server_key: "c92102dd1ed9e6e6b355f43dddbbd092"
server_key_iv: "3b44432039f84198c84f0d84333dc06d"
server_key_version: "encryptionkey@0"
I20260929 18:35:14.127513 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:14.144001 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:14.147583 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:14.147801 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:14.155084 21072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46807
I20260929 18:35:14.155107 21139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46807 every 8 connection(s)
I20260929 18:35:14.156069 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:14.158195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21072
I20260929 18:35:14.158347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:14.158670 392 external_mini_cluster.cc:1428] Setting key e30b28f734f3cccc997fde17f791fab8
I20260929 18:35:14.159195 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:14.165369 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e: Bootstrap starting.
I20260929 18:35:14.168243 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:14.160113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33221 (local address 127.0.0.1:46807)
0929 18:35:14.160598 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:14.160615 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:14.160651 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:14.161339 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:14.161356 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:14.161699 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:14.162051 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:14.162453 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:14.163437 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:14.164424 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:14.164924 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:14.165119 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:14.166073 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:14.166091 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:14.166108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:14.166616 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:14.166697 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:14.166741 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:14.166848 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:14.167021 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:35:14.167254 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":155,"threads_started":1}
I20260929 18:35:14.168393 21140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:14.169257 21140 log.cc:824] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:14.171551 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e: No bootstrap required, opened a new log
I20260929 18:35:14.171618 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:14.175457 21140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260929 18:35:14.175704 21140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:14.175776 21140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc5a9a932a745c0a20b50e21ac28c5e, State: Initialized, Role: FOLLOWER
I20260929 18:35:14.176318 21140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260929 18:35:14.176448 21140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:14.176533 21140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:14.176662 21140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:14.177829 21140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260929 18:35:14.178013 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:14.173882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35647 (local address 127.0.0.1:46807)
0929 18:35:14.174033 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:14.174038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:14.174147 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:35:14.174296 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:14.174299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:14.174347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:14.174431 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:14.174709 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:14.175238 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:14.176256 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:14.176613 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:14.176735 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:14.177555 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:14.177559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:14.177562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:14.177575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:14.177608 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:14.177620 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:14.177698 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:14.177817 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:35:14.177873 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:14.178261 21140 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc5a9a932a745c0a20b50e21ac28c5e; no voters:
I20260929 18:35:14.178646 21140 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:14.178719 21150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:14.178920 21150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [term 1 LEADER]: Becoming Leader. State: Replica: 5bc5a9a932a745c0a20b50e21ac28c5e, State: Running, Role: LEADER
I20260929 18:35:14.179291 21150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260929 18:35:14.180024 21140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:14.180531 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc5a9a932a745c0a20b50e21ac28c5e. Latest consensus state: current_term: 1 leader_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260929 18:35:14.180701 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:14.181010 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc5a9a932a745c0a20b50e21ac28c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260929 18:35:14.181115 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:14.181187 21155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:14.184638 21155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:14.190922 21155 catalog_manager.cc:1347] Generated new cluster ID: ab1ff094372545dca5876678414a3021
I20260929 18:35:14.191039 21155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:14.200448 21155 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:14.201442 21155 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:14.207603 21155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc5a9a932a745c0a20b50e21ac28c5e: Generated new TSK 0
I20260929 18:35:14.208513 21155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:14.215873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21072
2026-09-29T18:35:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:14 dist-test-slave-fmv8 krb5kdc[21171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:14 dist-test-slave-fmv8 krb5kdc[21171](info): set up 2 sockets
Sep 29 18:35:14 dist-test-slave-fmv8 krb5kdc[21171](info): commencing operation
krb5kdc: starting...
W20260929 18:35:15.293205 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:15 dist-test-slave-fmv8 krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:15.419092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:15.534663 21184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:15.534945 21184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:15.534987 21184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:15.539162 21184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:15.539240 21184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:15.539260 21184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:15.539278 21184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:15.539294 21184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:15.544857 21184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:15.545990 21184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:15.547045 21184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:15.555832 21190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:15.555860 21189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:15.555986 21184 server_base.cc:1034] running on GCE node
W20260929 18:35:15.555861 21192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:15.556753 21184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:15.557910 21184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:15.559108 21184 hybrid_clock.cc:648] HybridClock initialized: now 1790706915559077 us; error 47 us; skew 500 ppm
Sep 29 18:35:15 dist-test-slave-fmv8 krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:15.561985 21184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:15.562965 21184 webserver.cc:460] Webserver started at http://127.0.0.1:40263/ using document root <none> and password file <none>
I20260929 18:35:15.563548 21184 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:15.563640 21184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:15.563865 21184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:15.566090 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "f78fe1d31d914f59b79dbb21bc0d8592"
format_stamp: "Formatted at 2026-09-29 18:35:15 on dist-test-slave-fmv8"
server_key: "7233ac81ad3aa61a5b0c3a4cbbab9acc"
server_key_iv: "c8e8708f2f552b66a93cbc4ce3e72453"
server_key_version: "encryptionkey@0"
I20260929 18:35:15.566720 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "f78fe1d31d914f59b79dbb21bc0d8592"
format_stamp: "Formatted at 2026-09-29 18:35:15 on dist-test-slave-fmv8"
server_key: "7233ac81ad3aa61a5b0c3a4cbbab9acc"
server_key_iv: "c8e8708f2f552b66a93cbc4ce3e72453"
server_key_version: "encryptionkey@0"
I20260929 18:35:15.570735 21184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:15.573513 21199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:15.575070 21184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:35:15.575227 21184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "f78fe1d31d914f59b79dbb21bc0d8592"
format_stamp: "Formatted at 2026-09-29 18:35:15 on dist-test-slave-fmv8"
server_key: "7233ac81ad3aa61a5b0c3a4cbbab9acc"
server_key_iv: "c8e8708f2f552b66a93cbc4ce3e72453"
server_key_version: "encryptionkey@0"
I20260929 18:35:15.575354 21184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:15.580199 21184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:15.583379 21184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:15.583602 21184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:15.591354 21184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39027
I20260929 18:35:15.591372 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39027 every 8 connection(s)
I20260929 18:35:15.592459 21184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:15.595404 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:15.595574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21184
I20260929 18:35:15.595734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:15.596031 392 external_mini_cluster.cc:1428] Setting key 581986ab87108c30712610669181b0e6
I20260929 18:35:15.601336 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:15.597435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51711 (local address 127.0.0.1:39027)
0929 18:35:15.597898 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:15.597913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:15.597949 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:15.598687 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:15.598705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:15.598988 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:15.599351 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:15.600109 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:15.600114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:15.600127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:15.600137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:15.600249 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:15.600270 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:15.600421 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:15.600491 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:35:15.600656 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260929 18:35:15.602308 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592: Bootstrap starting.
I20260929 18:35:15.604224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:15.604813 21252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:15.605575 21252 log.cc:824] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:15.607853 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592: No bootstrap required, opened a new log
I20260929 18:35:15.608094 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:15.606292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48647 (local address 127.0.0.1:39027)
0929 18:35:15.606400 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:15.606404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:15.606676 (+ 272us) server_negotiation.cc:407] Connection header received
0929 18:35:15.606803 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:15.606805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:15.606833 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:15.606892 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:15.607630 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:15.607634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:15.607636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:15.607681 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:15.607706 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:15.607717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:15.607791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:15.607947 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:35:15.607991 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:35:15.611887 21252 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260929 18:35:15.612123 21252 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:15.612224 21252 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78fe1d31d914f59b79dbb21bc0d8592, State: Initialized, Role: FOLLOWER
I20260929 18:35:15.612776 21252 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260929 18:35:15.612959 21252 raft_consensus.cc:388] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:15.613034 21252 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:15.613158 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:15.614194 21252 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260929 18:35:15.614552 21252 leader_election.cc:302] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f78fe1d31d914f59b79dbb21bc0d8592; no voters:
I20260929 18:35:15.614907 21252 leader_election.cc:288] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:15.615090 21262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:15.615361 21262 raft_consensus.cc:686] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [term 1 LEADER]: Becoming Leader. State: Replica: f78fe1d31d914f59b79dbb21bc0d8592, State: Running, Role: LEADER
I20260929 18:35:15.615780 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260929 18:35:15.616276 21252 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:15.617499 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } } }
I20260929 18:35:15.617640 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:15.618084 21264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f78fe1d31d914f59b79dbb21bc0d8592. Latest consensus state: current_term: 1 leader_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78fe1d31d914f59b79dbb21bc0d8592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } } }
I20260929 18:35:15.618202 21264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:15.618088 21271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:15.621132 21271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:15.626777 21271 catalog_manager.cc:1347] Generated new cluster ID: c27d9652df514bd382c1106a3fdcfa1e
I20260929 18:35:15.626863 21271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:15.650905 21271 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:15.651281 21271 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:15.658665 21271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f78fe1d31d914f59b79dbb21bc0d8592: Generated new TSK 0
I20260929 18:35:15.659396 21271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:15.666155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21184
2026-09-29T18:35:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:15 dist-test-slave-fmv8 krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:15 dist-test-slave-fmv8 krb5kdc[21283](info): set up 2 sockets
Sep 29 18:35:15 dist-test-slave-fmv8 krb5kdc[21283](info): commencing operation
krb5kdc: starting...
W20260929 18:35:16.716497 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:16 dist-test-slave-fmv8 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:16.843060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:16.958038 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:16.958410 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:16.958500 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:16.962663 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:16.962777 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:16.962814 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:16.962857 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:16.962890 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:16.968487 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:16.969642 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:16.970719 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:16.980211 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:16.980348 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:16.980748 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:16.980867 21296 server_base.cc:1034] running on GCE node
I20260929 18:35:16.981536 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:16.982740 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:16.983963 21296 hybrid_clock.cc:648] HybridClock initialized: now 1790706916983874 us; error 97 us; skew 500 ppm
Sep 29 18:35:16 dist-test-slave-fmv8 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:16.987205 21296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:16.988276 21296 webserver.cc:460] Webserver started at http://127.0.0.1:35693/ using document root <none> and password file <none>
I20260929 18:35:16.988880 21296 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:16.988973 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:16.989216 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:16.991320 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "af0b77a6d92f4f4990df05e1be8f64b4"
format_stamp: "Formatted at 2026-09-29 18:35:16 on dist-test-slave-fmv8"
server_key: "be62f78b3ba5faeefbd9fb99a7cc460e"
server_key_iv: "57e2b0de2db702608ea4ff0d7aa6c57d"
server_key_version: "encryptionkey@0"
I20260929 18:35:16.991922 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "af0b77a6d92f4f4990df05e1be8f64b4"
format_stamp: "Formatted at 2026-09-29 18:35:16 on dist-test-slave-fmv8"
server_key: "be62f78b3ba5faeefbd9fb99a7cc460e"
server_key_iv: "57e2b0de2db702608ea4ff0d7aa6c57d"
server_key_version: "encryptionkey@0"
I20260929 18:35:16.995899 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:16.998485 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:16.999750 21296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:16.999895 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "af0b77a6d92f4f4990df05e1be8f64b4"
format_stamp: "Formatted at 2026-09-29 18:35:16 on dist-test-slave-fmv8"
server_key: "be62f78b3ba5faeefbd9fb99a7cc460e"
server_key_iv: "57e2b0de2db702608ea4ff0d7aa6c57d"
server_key_version: "encryptionkey@0"
I20260929 18:35:17.000016 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:17.022912 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:17.026546 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:17.026770 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:17.034435 21296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35035
I20260929 18:35:17.034456 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35035 every 8 connection(s)
I20260929 18:35:17.035455 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:17.038364 21364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:17.040122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21296
I20260929 18:35:17.040423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:17.040736 392 external_mini_cluster.cc:1428] Setting key 9448dda1118fd0c4d1f3d1b38de66c24
I20260929 18:35:17.045186 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: Bootstrap starting.
I20260929 18:35:17.047474 21364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:17.048454 21364 log.cc:824] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:35:17 dist-test-slave-fmv8 krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:35:17.050680 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: No bootstrap required, opened a new log
I20260929 18:35:17.054569 21364 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } }
I20260929 18:35:17.054837 21364 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:17.054944 21364 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0b77a6d92f4f4990df05e1be8f64b4, State: Initialized, Role: FOLLOWER
I20260929 18:35:17.055465 21364 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } }
I20260929 18:35:17.055625 21364 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:17.055714 21364 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:17.055811 21364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:17.056083 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:17.042204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:57589 (local address 127.0.0.1:35035)
0929 18:35:17.042726 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:17.042743 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:17.042780 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:35:17.043481 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:17.043500 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:17.043871 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:17.044249 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:17.045121 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:17.046168 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:17.047088 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:17.047611 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:17.047759 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:17.050742 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:17.050749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:17.050769 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:17.050809 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:17.052751 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:17.053361 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:17.053367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:17.053371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:17.053490 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:17.053876 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:17.053881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:17.053888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:17.054305 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:17.054568 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:17.054873 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 18:35:17.055128 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":152,"threads_started":1}
I20260929 18:35:17.057070 21364 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } }
I20260929 18:35:17.057441 21364 leader_election.cc:302] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0b77a6d92f4f4990df05e1be8f64b4; no voters:
I20260929 18:35:17.057772 21364 leader_election.cc:288] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:17.057917 21369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:17.058153 21369 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [term 1 LEADER]: Becoming Leader. State: Replica: af0b77a6d92f4f4990df05e1be8f64b4, State: Running, Role: LEADER
I20260929 18:35:17.058504 21369 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } }
I20260929 18:35:17.059090 21364 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:17.059720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:17.060321 21370 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } } }
I20260929 18:35:17.060448 21370 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:17.060773 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0b77a6d92f4f4990df05e1be8f64b4. Latest consensus state: current_term: 1 leader_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0b77a6d92f4f4990df05e1be8f64b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35035 } } }
I20260929 18:35:17.060855 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:17.064220 21385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:17.064311 21385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:17.064384 21378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:17.065285 21378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:17.072302 21378 catalog_manager.cc:1347] Generated new cluster ID: 221c01da9b4042398712617e8c442dc3
I20260929 18:35:17.072388 21378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:17.074452 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:17.064853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:42573 (local address 127.0.0.1:35035)
0929 18:35:17.064989 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:17.064994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:17.065194 (+ 200us) server_negotiation.cc:407] Connection header received
0929 18:35:17.065323 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:17.065325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:17.065398 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:17.065486 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:17.066324 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:17.066903 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:17.068003 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:17.068361 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:17.068468 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:17.070291 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:17.070295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:17.070299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:17.070329 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:17.072141 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:17.072804 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:17.072808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:17.072810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:17.072897 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:17.073322 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:17.073325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:17.073327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:17.073564 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:17.073716 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:17.074163 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 18:35:17.074272 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:35:17.084374 21378 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:17.085667 21378 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:17.090652 21378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0b77a6d92f4f4990df05e1be8f64b4: Generated new TSK 0
I20260929 18:35:17.091516 21378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:17.101588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21296
W20260929 18:35:17.107224 21281 connection.cc:535] client connection to 127.0.0.1:35035 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35035): Connection reset by peer (error 104)
2026-09-29T18:35:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:17 dist-test-slave-fmv8 krb5kdc[21395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:17 dist-test-slave-fmv8 krb5kdc[21395](info): set up 2 sockets
Sep 29 18:35:17 dist-test-slave-fmv8 krb5kdc[21395](info): commencing operation
krb5kdc: starting...
W20260929 18:35:18.140517 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:18 dist-test-slave-fmv8 krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:18.265664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:18.378480 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:18.378829 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:18.378906 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:18.383036 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:18.383137 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:18.383175 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:18.383235 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:18.383275 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:18.388757 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:18.389935 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:18.391029 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:18.400291 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:18.400492 21408 server_base.cc:1034] running on GCE node
W20260929 18:35:18.400352 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:18.400291 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:18.401178 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:18.402326 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:18.403502 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790706918403471 us; error 45 us; skew 500 ppm
Sep 29 18:35:18 dist-test-slave-fmv8 krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:18.410378 21408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:18.411368 21408 webserver.cc:460] Webserver started at http://127.0.0.1:42315/ using document root <none> and password file <none>
I20260929 18:35:18.411965 21408 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:18.412063 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:18.412339 21408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:18.414505 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "4efad1e44bf644259fabb021fef0c639"
format_stamp: "Formatted at 2026-09-29 18:35:18 on dist-test-slave-fmv8"
server_key: "ef53aa382e77f7705b926a7853ae54f0"
server_key_iv: "a5041c0696832ed2c7eff1637a1b9caf"
server_key_version: "encryptionkey@0"
I20260929 18:35:18.415500 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "4efad1e44bf644259fabb021fef0c639"
format_stamp: "Formatted at 2026-09-29 18:35:18 on dist-test-slave-fmv8"
server_key: "ef53aa382e77f7705b926a7853ae54f0"
server_key_iv: "a5041c0696832ed2c7eff1637a1b9caf"
server_key_version: "encryptionkey@0"
I20260929 18:35:18.419492 21408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:18.422287 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:18.423601 21408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:18.423772 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4efad1e44bf644259fabb021fef0c639"
format_stamp: "Formatted at 2026-09-29 18:35:18 on dist-test-slave-fmv8"
server_key: "ef53aa382e77f7705b926a7853ae54f0"
server_key_iv: "a5041c0696832ed2c7eff1637a1b9caf"
server_key_version: "encryptionkey@0"
I20260929 18:35:18.423899 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:18.449994 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:18.453691 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:18.453917 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:18.461385 21408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33775
I20260929 18:35:18.461406 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33775 every 8 connection(s)
I20260929 18:35:18.462407 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:18.465262 21476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:18.471215 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: Bootstrap starting.
I20260929 18:35:18.472712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21408
I20260929 18:35:18.472859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:18.473151 392 external_mini_cluster.cc:1428] Setting key c5798012045ddd5a71b8405279847eda
I20260929 18:35:18.474169 21476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:18.475077 21476 log.cc:824] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:18.477330 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: No bootstrap required, opened a new log
I20260929 18:35:18.481362 21476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260929 18:35:18.481650 21476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:18.481736 21476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efad1e44bf644259fabb021fef0c639, State: Initialized, Role: FOLLOWER
I20260929 18:35:18.482322 21476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260929 18:35:18.482501 21476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:18.482589 21476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:18.482715 21476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:18.482918 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:18.474388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:60067 (local address 127.0.0.1:33775)
0929 18:35:18.474856 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:18.474871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:18.474907 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:18.475741 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:18.475768 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:18.476262 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:18.476621 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:18.477055 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:18.478104 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:18.479042 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:18.479553 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:18.479736 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:18.480912 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:18.480918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:18.480932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:18.481344 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:18.481400 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:18.481423 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:18.481588 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:18.481827 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:35:18.481968 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260929 18:35:18.483942 21476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260929 18:35:18.484345 21476 leader_election.cc:302] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efad1e44bf644259fabb021fef0c639; no voters:
I20260929 18:35:18.484654 21476 leader_election.cc:288] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:18.484721 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:18.484922 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [term 1 LEADER]: Becoming Leader. State: Replica: 4efad1e44bf644259fabb021fef0c639, State: Running, Role: LEADER
I20260929 18:35:18.485312 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } }
I20260929 18:35:18.485961 21476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:18.486120 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:18.489365 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efad1e44bf644259fabb021fef0c639. Latest consensus state: current_term: 1 leader_uuid: "4efad1e44bf644259fabb021fef0c639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } } }
I20260929 18:35:18.489509 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:18.491253 21499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:18.491367 21499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:18.491583 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4efad1e44bf644259fabb021fef0c639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efad1e44bf644259fabb021fef0c639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33775 } } }
I20260929 18:35:18.491688 21503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:18.491752 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:18.492838 21503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:18.492861 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:18.488519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:59107 (local address 127.0.0.1:33775)
0929 18:35:18.488624 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:18.488628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:18.488997 (+ 369us) server_negotiation.cc:407] Connection header received
0929 18:35:18.489122 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:18.489125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:18.489173 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:18.489258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:18.489600 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:18.490189 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:18.490994 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:18.491239 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:18.491351 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:18.492342 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:18.492346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:18.492349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:18.492363 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:18.492393 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:18.492412 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:18.492505 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:18.492689 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:35:18.492725 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:35:18.498698 21503 catalog_manager.cc:1347] Generated new cluster ID: 877b1f4cff8b4aeea79a395a775022fe
I20260929 18:35:18.498782 21503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:18.513438 21503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:18.514564 21503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:18.520922 21503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4efad1e44bf644259fabb021fef0c639: Generated new TSK 0
I20260929 18:35:18.521704 21503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:18.526070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21408
W20260929 18:35:18.531721 21393 connection.cc:535] client connection to 127.0.0.1:33775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33775): Connection reset by peer (error 104)
2026-09-29T18:35:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:18 dist-test-slave-fmv8 krb5kdc[21507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:18 dist-test-slave-fmv8 krb5kdc[21507](info): set up 2 sockets
Sep 29 18:35:18 dist-test-slave-fmv8 krb5kdc[21507](info): commencing operation
krb5kdc: starting...
W20260929 18:35:19.580425 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:19 dist-test-slave-fmv8 krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:19.707612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:19.822736 21520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:19.823176 21520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:19.823271 21520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:19.827674 21520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:19.827805 21520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:19.827852 21520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:19.827890 21520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:19.827926 21520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:19.833518 21520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:19.834726 21520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:19.835829 21520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:19.845480 21528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:19.845480 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:19.845607 21520 server_base.cc:1034] running on GCE node
W20260929 18:35:19.845602 21526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:19.846493 21520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:19.847746 21520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:19.848940 21520 hybrid_clock.cc:648] HybridClock initialized: now 1790706919848920 us; error 35 us; skew 500 ppm
Sep 29 18:35:19 dist-test-slave-fmv8 krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:19.852236 21520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:19.853340 21520 webserver.cc:460] Webserver started at http://127.0.0.1:43153/ using document root <none> and password file <none>
I20260929 18:35:19.853957 21520 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:19.854056 21520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:19.854310 21520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:19.856585 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa"
format_stamp: "Formatted at 2026-09-29 18:35:19 on dist-test-slave-fmv8"
server_key: "79877dd253aa7bb463a9c8306eae1b50"
server_key_iv: "8555d9da65ff876c83d06dec66195dff"
server_key_version: "encryptionkey@0"
I20260929 18:35:19.857157 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa"
format_stamp: "Formatted at 2026-09-29 18:35:19 on dist-test-slave-fmv8"
server_key: "79877dd253aa7bb463a9c8306eae1b50"
server_key_iv: "8555d9da65ff876c83d06dec66195dff"
server_key_version: "encryptionkey@0"
I20260929 18:35:19.861078 21520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:19.863782 21535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:19.865302 21520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:19.865465 21520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa"
format_stamp: "Formatted at 2026-09-29 18:35:19 on dist-test-slave-fmv8"
server_key: "79877dd253aa7bb463a9c8306eae1b50"
server_key_iv: "8555d9da65ff876c83d06dec66195dff"
server_key_version: "encryptionkey@0"
I20260929 18:35:19.865600 21520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:19.870457 21520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:19.873876 21520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:19.874080 21520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:19.881677 21520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46613
I20260929 18:35:19.881695 21587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46613 every 8 connection(s)
I20260929 18:35:19.882694 21520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:19.884114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21520
I20260929 18:35:19.884270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:19.884593 392 external_mini_cluster.cc:1428] Setting key 53ad57f87980519e4983e21a4484317a
I20260929 18:35:19.885946 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:19.889906 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:19.886070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:50721 (local address 127.0.0.1:46613)
0929 18:35:19.886513 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:19.886529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:19.886566 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:35:19.887187 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:19.887202 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:19.887516 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:19.887861 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:19.888711 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:19.888717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:19.888736 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:19.888746 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:19.888799 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:19.888819 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:19.888968 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:19.889039 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:35:19.889195 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":142,"threads_started":1}
I20260929 18:35:19.892352 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa: Bootstrap starting.
I20260929 18:35:19.893020 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:19.894816 21588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:19.895777 21588 log.cc:824] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:19.896848 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:19.895158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:44515 (local address 127.0.0.1:46613)
0929 18:35:19.895289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:19.895293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:19.895451 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:35:19.895562 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:19.895567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:19.895604 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:19.895690 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:19.896411 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:19.896415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:19.896417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:19.896431 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:19.896460 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:19.896474 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:19.896571 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:19.896679 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:35:19.896713 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:35:19.897987 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa: No bootstrap required, opened a new log
I20260929 18:35:19.901551 21588 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } }
I20260929 18:35:19.901823 21588 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:19.901916 21588 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd841e365b2c43fc9bc24b08aa8c4aaa, State: Initialized, Role: FOLLOWER
I20260929 18:35:19.902475 21588 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } }
I20260929 18:35:19.902657 21588 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:19.902736 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:19.902849 21588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:19.903889 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } }
I20260929 18:35:19.904301 21588 leader_election.cc:302] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd841e365b2c43fc9bc24b08aa8c4aaa; no voters:
I20260929 18:35:19.904627 21588 leader_election.cc:288] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:19.904758 21598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:19.904994 21598 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [term 1 LEADER]: Becoming Leader. State: Replica: cd841e365b2c43fc9bc24b08aa8c4aaa, State: Running, Role: LEADER
I20260929 18:35:19.905335 21598 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } }
I20260929 18:35:19.905939 21588 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:19.907133 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } } }
I20260929 18:35:19.907199 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd841e365b2c43fc9bc24b08aa8c4aaa. Latest consensus state: current_term: 1 leader_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd841e365b2c43fc9bc24b08aa8c4aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46613 } } }
I20260929 18:35:19.907279 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:19.907301 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:19.907651 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:19.911046 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:19.917712 21607 catalog_manager.cc:1347] Generated new cluster ID: a46551f4c1b4428e962f414b815f07a5
I20260929 18:35:19.917800 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:19.936823 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:19.937538 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:19.953735 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd841e365b2c43fc9bc24b08aa8c4aaa: Generated new TSK 0
I20260929 18:35:19.954741 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:19.966012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21520
2026-09-29T18:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:20 dist-test-slave-fmv8 krb5kdc[21619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:20 dist-test-slave-fmv8 krb5kdc[21619](info): set up 2 sockets
Sep 29 18:35:20 dist-test-slave-fmv8 krb5kdc[21619](info): commencing operation
krb5kdc: starting...
W20260929 18:35:21.008472 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:21.145247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:21.258692 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:21.259095 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:21.259178 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:21.263387 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:21.263513 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:21.263553 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:21.263626 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:21.263669 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:21.269271 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:21.270459 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:21.271512 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:21.280228 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:21.280228 21638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:21.280231 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:21.280580 21632 server_base.cc:1034] running on GCE node
I20260929 18:35:21.281175 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:21.282399 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:21.283614 21632 hybrid_clock.cc:648] HybridClock initialized: now 1790706921283577 us; error 33 us; skew 500 ppm
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:21.286481 21632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:21.287531 21632 webserver.cc:460] Webserver started at http://127.0.0.1:36123/ using document root <none> and password file <none>
I20260929 18:35:21.288213 21632 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:21.288308 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:21.288551 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:21.290665 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "4d8d3c2e52db415fbfcff9c25f7c8705"
format_stamp: "Formatted at 2026-09-29 18:35:21 on dist-test-slave-fmv8"
server_key: "e67611735f233c60bcadc590039d915a"
server_key_iv: "4ccf8de08f87c20d2e511d2d7880ca90"
server_key_version: "encryptionkey@0"
I20260929 18:35:21.291263 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d8d3c2e52db415fbfcff9c25f7c8705"
format_stamp: "Formatted at 2026-09-29 18:35:21 on dist-test-slave-fmv8"
server_key: "e67611735f233c60bcadc590039d915a"
server_key_iv: "4ccf8de08f87c20d2e511d2d7880ca90"
server_key_version: "encryptionkey@0"
I20260929 18:35:21.295260 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:21.297916 21647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:21.299249 21632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:35:21.299459 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "4d8d3c2e52db415fbfcff9c25f7c8705"
format_stamp: "Formatted at 2026-09-29 18:35:21 on dist-test-slave-fmv8"
server_key: "e67611735f233c60bcadc590039d915a"
server_key_iv: "4ccf8de08f87c20d2e511d2d7880ca90"
server_key_version: "encryptionkey@0"
I20260929 18:35:21.299594 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:21.326586 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:21.330103 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:21.330339 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:21.337796 21632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37551
I20260929 18:35:21.337811 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37551 every 8 connection(s)
I20260929 18:35:21.338788 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:21.341593 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:21.342185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21632
I20260929 18:35:21.342338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:21.342661 392 external_mini_cluster.cc:1428] Setting key cc5c3b597509164a9687efba29b7bb70
I20260929 18:35:21.348673 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: Bootstrap starting.
I20260929 18:35:21.351207 21700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: Neither blocks nor log segments found. Creating new log.
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:35:21.351997 21700 log.cc:824] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:21.354096 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: No bootstrap required, opened a new log
I20260929 18:35:21.357331 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:21.344066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:38109 (local address 127.0.0.1:37551)
0929 18:35:21.344567 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:21.344583 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:21.344620 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:35:21.345291 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:21.345309 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:21.345663 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:21.346030 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:21.346881 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:21.347883 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:21.348921 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:21.349591 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:21.349760 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:21.352522 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:21.352527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:21.352541 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:21.352567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:21.354447 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:21.354975 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:21.354980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:21.354985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:21.355094 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:21.355461 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:21.355465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:21.355467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:21.355852 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:21.356091 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:21.356379 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 18:35:21.356606 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":135,"threads_started":1}
I20260929 18:35:21.357868 21700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260929 18:35:21.358120 21700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:21.358209 21700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8d3c2e52db415fbfcff9c25f7c8705, State: Initialized, Role: FOLLOWER
I20260929 18:35:21.358733 21700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260929 18:35:21.358896 21700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:21.358950 21700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:21.359046 21700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:21.360273 21700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260929 18:35:21.360635 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:21.360633 21700 leader_election.cc:302] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d8d3c2e52db415fbfcff9c25f7c8705; no voters:
I20260929 18:35:21.360976 21700 leader_election.cc:288] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:21.361084 21705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:21.361320 21705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8d3c2e52db415fbfcff9c25f7c8705, State: Running, Role: LEADER
I20260929 18:35:21.361711 21705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260929 18:35:21.362323 21700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:21.363402 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260929 18:35:21.363549 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:21.363945 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8d3c2e52db415fbfcff9c25f7c8705. Latest consensus state: current_term: 1 leader_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8d3c2e52db415fbfcff9c25f7c8705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260929 18:35:21.364087 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:21.367120 21725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:21.367218 21725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:21.367332 21719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:21.368269 21719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:21.373622 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:21.363208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:57689 (local address 127.0.0.1:37551)
0929 18:35:21.363313 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:21.363317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:21.364467 (+ 1150us) server_negotiation.cc:407] Connection header received
0929 18:35:21.364803 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:21.364807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:21.364860 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:21.364947 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:21.365659 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:21.366411 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:21.367574 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:21.367944 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:21.368060 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:21.369767 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:21.369771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:21.369773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:21.369810 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:21.371361 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:21.371852 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:21.371855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:21.371859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:21.371951 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:21.372354 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:21.372357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:21.372359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:21.372544 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:21.372718 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:21.373369 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 18:35:21.373475 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:35:21.374981 21719 catalog_manager.cc:1347] Generated new cluster ID: 83de2ae4b5f24128865dfa0656eeb13f
I20260929 18:35:21.375051 21719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:21.384349 21719 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:21.385597 21719 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:21.393159 21719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d8d3c2e52db415fbfcff9c25f7c8705: Generated new TSK 0
I20260929 18:35:21.393993 21719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:21.401976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21632
W20260929 18:35:21.407300 21617 connection.cc:535] client connection to 127.0.0.1:37551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37551): Connection reset by peer (error 104)
2026-09-29T18:35:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21731](info): set up 2 sockets
Sep 29 18:35:21 dist-test-slave-fmv8 krb5kdc[21731](info): commencing operation
krb5kdc: starting...
W20260929 18:35:22.444399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:22 dist-test-slave-fmv8 krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:22.566689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:22.685154 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:22.685441 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:22.685483 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:22.689574 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:22.689636 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:22.689656 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:22.689672 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:22.689719 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:22.695070 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:22.696194 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:22.697220 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:22.705618 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:22.705629 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:22.705717 21744 server_base.cc:1034] running on GCE node
W20260929 18:35:22.705657 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:22.706401 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:22.707448 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:22.708644 21744 hybrid_clock.cc:648] HybridClock initialized: now 1790706922708613 us; error 43 us; skew 500 ppm
Sep 29 18:35:22 dist-test-slave-fmv8 krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:22.711508 21744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:22.712494 21744 webserver.cc:460] Webserver started at http://127.0.0.1:39397/ using document root <none> and password file <none>
I20260929 18:35:22.713096 21744 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:22.713193 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:22.713434 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:22.715447 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "98bda1ad8f0644058bf9644e53cb375e"
format_stamp: "Formatted at 2026-09-29 18:35:22 on dist-test-slave-fmv8"
server_key: "59ea8412c0cf57147d49131286c42d08"
server_key_iv: "a7b78cd3343b24f4e060fb71d9693c38"
server_key_version: "encryptionkey@0"
I20260929 18:35:22.716019 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "98bda1ad8f0644058bf9644e53cb375e"
format_stamp: "Formatted at 2026-09-29 18:35:22 on dist-test-slave-fmv8"
server_key: "59ea8412c0cf57147d49131286c42d08"
server_key_iv: "a7b78cd3343b24f4e060fb71d9693c38"
server_key_version: "encryptionkey@0"
I20260929 18:35:22.719725 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:22.722319 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:22.723629 21744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:35:22.723796 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "98bda1ad8f0644058bf9644e53cb375e"
format_stamp: "Formatted at 2026-09-29 18:35:22 on dist-test-slave-fmv8"
server_key: "59ea8412c0cf57147d49131286c42d08"
server_key_iv: "a7b78cd3343b24f4e060fb71d9693c38"
server_key_version: "encryptionkey@0"
I20260929 18:35:22.723915 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:22.742866 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:22.746342 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:22.746554 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:22.753616 21744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34545
I20260929 18:35:22.753645 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34545 every 8 connection(s)
I20260929 18:35:22.754535 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:22.757239 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:22.762544 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e: Bootstrap starting.
I20260929 18:35:22.762982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21744
I20260929 18:35:22.763118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:22.763391 392 external_mini_cluster.cc:1428] Setting key 73c0ae38eae57d3e57633938acee0722
I20260929 18:35:22.765306 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:22.766183 21812 log.cc:824] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:22.768399 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e: No bootstrap required, opened a new log
I20260929 18:35:22.772660 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260929 18:35:22.772933 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:22.772989 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bda1ad8f0644058bf9644e53cb375e, State: Initialized, Role: FOLLOWER
I20260929 18:35:22.773489 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260929 18:35:22.773650 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:22.773710 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:22.773723 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:22.764740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:38559 (local address 127.0.0.1:34545)
0929 18:35:22.765194 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:22.765216 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:22.765262 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:35:22.766130 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:22.766154 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:22.766583 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:22.766916 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:22.767331 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:22.768830 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:22.769783 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:22.770501 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:22.770680 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:22.771616 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:22.771623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:22.771640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:22.772179 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:22.772248 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:22.772277 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:22.772408 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:22.772611 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:35:22.772746 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
I20260929 18:35:22.773821 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:22.774782 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260929 18:35:22.775194 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98bda1ad8f0644058bf9644e53cb375e; no voters:
I20260929 18:35:22.775539 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:22.775653 21817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:22.775943 21817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [term 1 LEADER]: Becoming Leader. State: Replica: 98bda1ad8f0644058bf9644e53cb375e, State: Running, Role: LEADER
I20260929 18:35:22.776356 21817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260929 18:35:22.776849 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:22.776968 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:22.778165 21818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bda1ad8f0644058bf9644e53cb375e. Latest consensus state: current_term: 1 leader_uuid: "98bda1ad8f0644058bf9644e53cb375e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } } }
I20260929 18:35:22.778158 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98bda1ad8f0644058bf9644e53cb375e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bda1ad8f0644058bf9644e53cb375e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } } }
I20260929 18:35:22.778321 21818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:22.778321 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:22.778851 21830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:22.782022 21830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:22.784101 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:22.779687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:57877 (local address 127.0.0.1:34545)
0929 18:35:22.779816 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:22.779821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:22.780370 (+ 549us) server_negotiation.cc:407] Connection header received
0929 18:35:22.780553 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:22.780559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:22.780621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:22.780696 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:22.780982 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:22.781499 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:22.782405 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:22.782663 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:22.782778 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:22.783646 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:22.783649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:22.783652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:22.783664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:22.783696 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:22.783706 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:22.783802 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:22.783923 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 18:35:22.783956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:35:22.787689 21830 catalog_manager.cc:1347] Generated new cluster ID: cedd9a5c642144ee862c7cca5cc9381c
I20260929 18:35:22.787760 21830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:22.799767 21830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:22.801010 21830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:22.808645 21830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98bda1ad8f0644058bf9644e53cb375e: Generated new TSK 0
I20260929 18:35:22.809429 21830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:22.817993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21744
W20260929 18:35:22.823321 21729 connection.cc:535] client connection to 127.0.0.1:34545 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34545): Connection reset by peer (error 104)
2026-09-29T18:35:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:22 dist-test-slave-fmv8 krb5kdc[21843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:22 dist-test-slave-fmv8 krb5kdc[21843](info): set up 2 sockets
Sep 29 18:35:22 dist-test-slave-fmv8 krb5kdc[21843](info): commencing operation
krb5kdc: starting...
W20260929 18:35:23.876952 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:23 dist-test-slave-fmv8 krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:35:24.001206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:24.113968 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:24.114308 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:24.114385 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:24.118551 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:24.118659 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:24.118696 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:24.118748 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:24.118793 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:24.124346 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:24.125510 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:24.126641 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:24.135536 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.135555 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.135525 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:24.135526 21856 server_base.cc:1034] running on GCE node
I20260929 18:35:24.136507 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:24.137770 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:24.138969 21856 hybrid_clock.cc:648] HybridClock initialized: now 1790706924138956 us; error 31 us; skew 500 ppm
Sep 29 18:35:24 dist-test-slave-fmv8 krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:24.146206 21856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:35:24.147280 21856 webserver.cc:460] Webserver started at http://127.0.0.1:37613/ using document root <none> and password file <none>
I20260929 18:35:24.147904 21856 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:24.148002 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:24.148319 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:24.150532 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "442fb3d4ba3f47fbb02ab233414d9395"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-fmv8"
server_key: "ae94bcec46534261fc1799fa77b5b39c"
server_key_iv: "a829c8d8ea9b536c9b5063032d24a6c8"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.151122 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "442fb3d4ba3f47fbb02ab233414d9395"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-fmv8"
server_key: "ae94bcec46534261fc1799fa77b5b39c"
server_key_iv: "a829c8d8ea9b536c9b5063032d24a6c8"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.155072 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:35:24.157997 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.159427 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:35:24.159592 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "442fb3d4ba3f47fbb02ab233414d9395"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-fmv8"
server_key: "ae94bcec46534261fc1799fa77b5b39c"
server_key_iv: "a829c8d8ea9b536c9b5063032d24a6c8"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.159729 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:24.164619 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:24.167829 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:24.168076 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:24.175740 21856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46801
I20260929 18:35:24.175773 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46801 every 8 connection(s)
I20260929 18:35:24.176836 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:24.178457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21856
I20260929 18:35:24.178628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:24.178938 392 external_mini_cluster.cc:1428] Setting key 84be96c66c79684bd63db3d05d9f99b6
I20260929 18:35:24.180192 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:24.184684 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.180494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:52613 (local address 127.0.0.1:46801)
0929 18:35:24.180937 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.180964 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.180999 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:35:24.181695 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.181713 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.182020 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.182321 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.183276 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.183280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.183293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:24.183303 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.183361 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:24.183381 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.183500 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.183632 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.183824 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":152,"threads_started":1}
I20260929 18:35:24.187322 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395: Bootstrap starting.
I20260929 18:35:24.187914 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:24.189949 21924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:24.190910 21924 log.cc:824] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:24.191953 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.190103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:48895 (local address 127.0.0.1:46801)
0929 18:35:24.190244 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.190251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.190495 (+ 244us) server_negotiation.cc:407] Connection header received
0929 18:35:24.190606 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.190609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.190647 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.190738 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.191483 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.191486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.191488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:24.191498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.191530 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:24.191540 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.191609 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.191783 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.191823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:35:24.193192 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395: No bootstrap required, opened a new log
I20260929 18:35:24.196822 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } }
I20260929 18:35:24.197075 21924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:24.197119 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fb3d4ba3f47fbb02ab233414d9395, State: Initialized, Role: FOLLOWER
I20260929 18:35:24.197667 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } }
I20260929 18:35:24.197876 21924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:24.197978 21924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:24.198089 21924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:24.199105 21924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } }
I20260929 18:35:24.199501 21924 leader_election.cc:302] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 442fb3d4ba3f47fbb02ab233414d9395; no voters:
I20260929 18:35:24.199839 21924 leader_election.cc:288] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:24.199994 21934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:24.200269 21934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [term 1 LEADER]: Becoming Leader. State: Replica: 442fb3d4ba3f47fbb02ab233414d9395, State: Running, Role: LEADER
I20260929 18:35:24.200636 21934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } }
I20260929 18:35:24.201201 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:24.202502 21935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "442fb3d4ba3f47fbb02ab233414d9395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } } }
I20260929 18:35:24.202638 21935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:24.202919 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 442fb3d4ba3f47fbb02ab233414d9395. Latest consensus state: current_term: 1 leader_uuid: "442fb3d4ba3f47fbb02ab233414d9395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442fb3d4ba3f47fbb02ab233414d9395" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46801 } } }
I20260929 18:35:24.203055 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:24.203114 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:24.206285 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:24.211862 21943 catalog_manager.cc:1347] Generated new cluster ID: 879540ce2296402c807f4be849b5d5c7
I20260929 18:35:24.211947 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:24.218261 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:24.218628 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:24.224875 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 442fb3d4ba3f47fbb02ab233414d9395: Generated new TSK 0
I20260929 18:35:24.225749 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:24.228960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21856
2026-09-29T18:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:24 dist-test-slave-fmv8 krb5kdc[21955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:24 dist-test-slave-fmv8 krb5kdc[21955](info): set up 2 sockets
Sep 29 18:35:24 dist-test-slave-fmv8 krb5kdc[21955](info): commencing operation
krb5kdc: starting...
W20260929 18:35:25.284795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:25.407696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:25.521286 21968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:25.521610 21968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:25.521703 21968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:25.525923 21968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:25.526012 21968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:25.526057 21968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:25.526116 21968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:25.526157 21968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:25.531418 21968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:25.532557 21968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:25.533653 21968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:25.542048 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:25.542047 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:25.542080 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:25.542390 21968 server_base.cc:1034] running on GCE node
I20260929 18:35:25.542917 21968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:25.544044 21968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:25.545262 21968 hybrid_clock.cc:648] HybridClock initialized: now 1790706925545233 us; error 41 us; skew 500 ppm
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:25.548007 21968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:25.548931 21968 webserver.cc:460] Webserver started at http://127.0.98.62:37297/ using document root <none> and password file <none>
I20260929 18:35:25.549521 21968 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:25.549616 21968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:25.549847 21968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:25.551826 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "46a388f582024329b1171578b3038e9c"
format_stamp: "Formatted at 2026-09-29 18:35:25 on dist-test-slave-fmv8"
server_key: "e3df2cbfc20624f5e2c3e070eb181a54"
server_key_iv: "ee41acb55ff2b09d1db789f57df23d28"
server_key_version: "encryptionkey@0"
I20260929 18:35:25.552429 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "46a388f582024329b1171578b3038e9c"
format_stamp: "Formatted at 2026-09-29 18:35:25 on dist-test-slave-fmv8"
server_key: "e3df2cbfc20624f5e2c3e070eb181a54"
server_key_iv: "ee41acb55ff2b09d1db789f57df23d28"
server_key_version: "encryptionkey@0"
I20260929 18:35:25.555987 21968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:25.558421 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:25.559674 21968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:25.559815 21968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "46a388f582024329b1171578b3038e9c"
format_stamp: "Formatted at 2026-09-29 18:35:25 on dist-test-slave-fmv8"
server_key: "e3df2cbfc20624f5e2c3e070eb181a54"
server_key_iv: "ee41acb55ff2b09d1db789f57df23d28"
server_key_version: "encryptionkey@0"
I20260929 18:35:25.559930 21968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:25.574054 21968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:25.577093 21968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:25.577307 21968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:25.584714 21968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43909
I20260929 18:35:25.584736 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43909 every 8 connection(s)
I20260929 18:35:25.585675 21968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:25.588438 22036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:25.593928 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c: Bootstrap starting.
I20260929 18:35:25.594002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 21968
I20260929 18:35:25.594139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:25.594424 392 external_mini_cluster.cc:1428] Setting key c9f50695e82c0edfc8e9ca5ac132307e
I20260929 18:35:25.596570 22036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:25.597286 22036 log.cc:824] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:25.599086 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c: No bootstrap required, opened a new log
I20260929 18:35:25.602625 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } }
I20260929 18:35:25.602910 22036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:25.602972 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a388f582024329b1171578b3038e9c, State: Initialized, Role: FOLLOWER
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[21955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:25.603462 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } }
I20260929 18:35:25.603631 22036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:25.603700 22036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:25.603806 22036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:25.604764 22036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } }
I20260929 18:35:25.605103 22036 leader_election.cc:302] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46a388f582024329b1171578b3038e9c; no voters:
I20260929 18:35:25.605413 22036 leader_election.cc:288] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:25.605536 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:25.605739 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [term 1 LEADER]: Becoming Leader. State: Replica: 46a388f582024329b1171578b3038e9c, State: Running, Role: LEADER
I20260929 18:35:25.606106 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } }
I20260929 18:35:25.606739 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:25.607800 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46a388f582024329b1171578b3038e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } } }
I20260929 18:35:25.607915 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:25.608280 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a388f582024329b1171578b3038e9c. Latest consensus state: current_term: 1 leader_uuid: "46a388f582024329b1171578b3038e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a388f582024329b1171578b3038e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43909 } } }
I20260929 18:35:25.608429 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:25.608536 22050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:25.610596 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.595782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42453 (local address 127.0.98.62:43909)
0929 18:35:25.596285 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:25.596301 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:25.596336 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:35:25.597066 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:25.597084 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:25.597484 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:25.597830 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:25.598666 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.599691 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.600642 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.601109 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.601278 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.604026 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:25.604031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:25.604043 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:25.604069 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:25.606266 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:25.606850 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:25.606856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:25.606861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:25.606996 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:25.607453 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:25.607458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:25.607461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:25.608042 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:25.609240 (+ 1198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:25.609424 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:35:25.609648 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":136,"threads_started":1}
I20260929 18:35:25.611743 22050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:25.613999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:25.617834 22050 catalog_manager.cc:1347] Generated new cluster ID: 28a92e5880f44f5295beec8fd067f926
I20260929 18:35:25.617894 22050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:25.625694 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.615979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33291 (local address 127.0.98.62:43909)
0929 18:35:25.616112 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:25.616117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:25.616350 (+ 233us) server_negotiation.cc:407] Connection header received
0929 18:35:25.616545 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:25.616549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:25.616601 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:25.616685 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:25.617629 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.618439 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.619643 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.619995 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.620088 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.621956 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:25.621960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:25.621963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:25.621989 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:25.623595 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:25.624222 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:25.624226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:25.624228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:25.624312 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:25.624702 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:25.624705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:25.624707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:25.624908 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:25.625021 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:25.625437 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 18:35:25.625530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:35:25.641237 22050 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:25.642233 22050 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:25.648780 22050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46a388f582024329b1171578b3038e9c: Generated new TSK 0
I20260929 18:35:25.649603 22050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:25.663173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 21968
W20260929 18:35:25.668403 21953 connection.cc:535] client connection to 127.0.98.62:43909 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43909): Connection reset by peer (error 104)
2026-09-29T18:35:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[22067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[22067](info): set up 2 sockets
Sep 29 18:35:25 dist-test-slave-fmv8 krb5kdc[22067](info): commencing operation
krb5kdc: starting...
W20260929 18:35:26.716499 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:26 dist-test-slave-fmv8 krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:26.838976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:26.954413 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:26.954708 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:26.954762 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:26.958981 22080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:26.959059 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:26.959080 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:26.959097 22080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:26.959112 22080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:26.964602 22080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:26.965740 22080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:26.966732 22080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:26.975634 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:26.975620 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:26.975723 22080 server_base.cc:1034] running on GCE node
W20260929 18:35:26.975620 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:26.976747 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:26.977936 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:26.979116 22080 hybrid_clock.cc:648] HybridClock initialized: now 1790706926979090 us; error 51 us; skew 500 ppm
Sep 29 18:35:26 dist-test-slave-fmv8 krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:26.982189 22080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:26.983230 22080 webserver.cc:460] Webserver started at http://127.0.98.62:44005/ using document root <none> and password file <none>
I20260929 18:35:26.983834 22080 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:26.983907 22080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:26.984129 22080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:26.986400 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "0df526b6c5a54f10816a786b8b9a0475"
format_stamp: "Formatted at 2026-09-29 18:35:26 on dist-test-slave-fmv8"
server_key: "f7076648009f06a87a1bf6860ea5d03c"
server_key_iv: "71e63040310da1a963668e3f32bd1e75"
server_key_version: "encryptionkey@0"
I20260929 18:35:26.987001 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "0df526b6c5a54f10816a786b8b9a0475"
format_stamp: "Formatted at 2026-09-29 18:35:26 on dist-test-slave-fmv8"
server_key: "f7076648009f06a87a1bf6860ea5d03c"
server_key_iv: "71e63040310da1a963668e3f32bd1e75"
server_key_version: "encryptionkey@0"
I20260929 18:35:26.991196 22080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:26.993791 22095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:26.995071 22080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:26.995172 22080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "0df526b6c5a54f10816a786b8b9a0475"
format_stamp: "Formatted at 2026-09-29 18:35:26 on dist-test-slave-fmv8"
server_key: "f7076648009f06a87a1bf6860ea5d03c"
server_key_iv: "71e63040310da1a963668e3f32bd1e75"
server_key_version: "encryptionkey@0"
I20260929 18:35:26.995308 22080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:27.023594 22080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:27.026878 22080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:27.027199 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:27.034233 22080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40399
I20260929 18:35:27.034258 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40399 every 8 connection(s)
I20260929 18:35:27.035189 22080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:27.036018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22080
I20260929 18:35:27.036197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:27.036500 392 external_mini_cluster.cc:1428] Setting key dd2d4c622ab52c825031dcac248ffa16
I20260929 18:35:27.038431 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:27.044950 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475: Bootstrap starting.
I20260929 18:35:27.046135 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:27.037922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48715 (local address 127.0.98.62:40399)
0929 18:35:27.038364 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:27.038378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:27.038414 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:27.039123 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:27.039141 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:27.039502 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:27.039788 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:27.040212 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:27.041222 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:27.042497 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:27.043143 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:27.043265 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:27.044289 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:27.044294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:27.044307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:27.044698 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:27.044756 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:27.044779 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:27.044881 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:27.045308 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 18:35:27.045435 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":160,"threads_started":1}
I20260929 18:35:27.047551 22148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:27.048434 22148 log.cc:824] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:27.049332 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:27.050450 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475: No bootstrap required, opened a new log
I20260929 18:35:27.055294 22148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260929 18:35:27.055584 22148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:27.055660 22148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df526b6c5a54f10816a786b8b9a0475, State: Initialized, Role: FOLLOWER
I20260929 18:35:27.055711 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:27.051322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58369 (local address 127.0.98.62:40399)
0929 18:35:27.051441 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:27.051446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:27.051690 (+ 244us) server_negotiation.cc:407] Connection header received
0929 18:35:27.051801 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:27.051803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:27.051840 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:27.051904 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:27.052265 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:27.052800 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:27.053962 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:27.054179 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:27.054272 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:27.055130 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:27.055133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:27.055134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:27.055145 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:27.055169 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:27.055179 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:27.055251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:27.055434 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:35:27.055497 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:27.056114 22148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260929 18:35:27.056326 22148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:27.056396 22148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:27.056490 22148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:27.057430 22148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260929 18:35:27.057776 22148 leader_election.cc:302] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df526b6c5a54f10816a786b8b9a0475; no voters:
I20260929 18:35:27.058136 22148 leader_election.cc:288] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:27.058252 22158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:27.058447 22158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [term 1 LEADER]: Becoming Leader. State: Replica: 0df526b6c5a54f10816a786b8b9a0475, State: Running, Role: LEADER
I20260929 18:35:27.058872 22158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260929 18:35:27.059363 22148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:27.060655 22160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df526b6c5a54f10816a786b8b9a0475. Latest consensus state: current_term: 1 leader_uuid: "0df526b6c5a54f10816a786b8b9a0475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
I20260929 18:35:27.060766 22160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:27.061086 22159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df526b6c5a54f10816a786b8b9a0475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df526b6c5a54f10816a786b8b9a0475" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
I20260929 18:35:27.061167 22168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:27.061184 22159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:27.064067 22168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:27.069173 22168 catalog_manager.cc:1347] Generated new cluster ID: dde274d6364d4c56a99f601e985c99f8
I20260929 18:35:27.069244 22168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:27.087824 22168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:27.088891 22168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:27.095841 22168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df526b6c5a54f10816a786b8b9a0475: Generated new TSK 0
I20260929 18:35:27.096649 22168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:27.103204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22080
W20260929 18:35:27.108577 22065 connection.cc:535] client connection to 127.0.98.62:40399 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40399): Connection reset by peer (error 104)
2026-09-29T18:35:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:27 dist-test-slave-fmv8 krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:27 dist-test-slave-fmv8 krb5kdc[22179](info): set up 2 sockets
Sep 29 18:35:27 dist-test-slave-fmv8 krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260929 18:35:28.136613 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:28 dist-test-slave-fmv8 krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:28.264098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:28.375919 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:28.376251 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:28.376345 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:28.380414 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:28.380501 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:28.380544 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:28.380601 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:28.380640 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:28.386330 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:28.387351 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:28.388384 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:28.396840 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:28.396840 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:28.396975 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:28.396996 22192 server_base.cc:1034] running on GCE node
I20260929 18:35:28.397639 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:28.398733 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:28.399926 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790706928399911 us; error 25 us; skew 500 ppm
Sep 29 18:35:28 dist-test-slave-fmv8 krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:28.402735 22192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:28.403613 22192 webserver.cc:460] Webserver started at http://127.0.98.62:39931/ using document root <none> and password file <none>
I20260929 18:35:28.404141 22192 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:28.404277 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:28.404505 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:28.406431 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "170de999d6564aff85c210a55f073c30"
format_stamp: "Formatted at 2026-09-29 18:35:28 on dist-test-slave-fmv8"
server_key: "b3b152bce9b28074d06ba711fe177bed"
server_key_iv: "1b35a54fe719f4ba713639f283a85475"
server_key_version: "encryptionkey@0"
I20260929 18:35:28.406971 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "170de999d6564aff85c210a55f073c30"
format_stamp: "Formatted at 2026-09-29 18:35:28 on dist-test-slave-fmv8"
server_key: "b3b152bce9b28074d06ba711fe177bed"
server_key_iv: "1b35a54fe719f4ba713639f283a85475"
server_key_version: "encryptionkey@0"
I20260929 18:35:28.410553 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:28.413015 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:28.414253 22192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:28.414402 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "170de999d6564aff85c210a55f073c30"
format_stamp: "Formatted at 2026-09-29 18:35:28 on dist-test-slave-fmv8"
server_key: "b3b152bce9b28074d06ba711fe177bed"
server_key_iv: "1b35a54fe719f4ba713639f283a85475"
server_key_version: "encryptionkey@0"
I20260929 18:35:28.414520 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:28.419119 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:28.422395 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:28.422580 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:28.429615 22192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40633
I20260929 18:35:28.429642 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40633 every 8 connection(s)
I20260929 18:35:28.430519 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:28.433136 22260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:28.438357 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: Bootstrap starting.
I20260929 18:35:28.440488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22192
I20260929 18:35:28.440634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:28.440828 22260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:28.440904 392 external_mini_cluster.cc:1428] Setting key 999b7896c398aa5efa418d3bd43d51c7
I20260929 18:35:28.441710 22260 log.cc:824] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:28.443681 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: No bootstrap required, opened a new log
I20260929 18:35:28.445840 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:28.442024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36029 (local address 127.0.98.62:40633)
0929 18:35:28.442498 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:28.442519 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:28.442573 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:35:28.443225 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:28.443242 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:28.443506 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:28.443807 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:28.444632 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:28.444636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:28.444647 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:28.444657 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:28.444707 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:28.444724 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:28.444808 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:28.444936 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:35:28.445166 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260929 18:35:28.448020 22260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } }
I20260929 18:35:28.448297 22260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:28.448365 22260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170de999d6564aff85c210a55f073c30, State: Initialized, Role: FOLLOWER
I20260929 18:35:28.448939 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:28.448939 22260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } }
I20260929 18:35:28.449116 22260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:28.449182 22260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:28.449277 22260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:28.450172 22260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } }
I20260929 18:35:28.450505 22260 leader_election.cc:302] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 170de999d6564aff85c210a55f073c30; no voters:
I20260929 18:35:28.450804 22260 leader_election.cc:288] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:28.450939 22269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:28.451162 22269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [term 1 LEADER]: Becoming Leader. State: Replica: 170de999d6564aff85c210a55f073c30, State: Running, Role: LEADER
I20260929 18:35:28.451511 22269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } }
I20260929 18:35:28.452045 22260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:28.453222 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:28.451281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60149 (local address 127.0.98.62:40633)
0929 18:35:28.451390 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:28.451395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:28.451643 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:35:28.451759 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:28.451762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:28.451795 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:28.451871 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:28.452701 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:28.452704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:28.452707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:28.452718 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:28.452745 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:28.452758 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:28.452819 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:28.453037 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 18:35:28.453083 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:35:28.453390 22272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "170de999d6564aff85c210a55f073c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } }
I20260929 18:35:28.453400 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170de999d6564aff85c210a55f073c30. Latest consensus state: current_term: 1 leader_uuid: "170de999d6564aff85c210a55f073c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170de999d6564aff85c210a55f073c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } }
I20260929 18:35:28.453601 22272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:28.453601 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:28.456768 22285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:28.456858 22285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:28.456954 22281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:28.457643 22281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:28.462764 22281 catalog_manager.cc:1347] Generated new cluster ID: d8a4435f151144b982fad42a66d37dd7
I20260929 18:35:28.462852 22281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:28.473958 22281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:28.474334 22281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:28.478674 22281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 170de999d6564aff85c210a55f073c30: Generated new TSK 0
I20260929 18:35:28.479390 22281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:28.489714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22192
2026-09-29T18:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:28 dist-test-slave-fmv8 krb5kdc[22291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:28 dist-test-slave-fmv8 krb5kdc[22291](info): set up 2 sockets
Sep 29 18:35:28 dist-test-slave-fmv8 krb5kdc[22291](info): commencing operation
krb5kdc: starting...
W20260929 18:35:29.528566 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:29.651250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39809
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:29.761507 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:29.761785 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:29.761827 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:29.765774 22304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:29.765829 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:29.765849 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:29.765865 22304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:29.765880 22304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:29.771057 22304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:29.772047 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:29.773132 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:29.781711 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:29.781736 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:29.781739 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:29.781966 22304 server_base.cc:1034] running on GCE node
I20260929 18:35:29.782516 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:29.783551 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:29.784734 22304 hybrid_clock.cc:648] HybridClock initialized: now 1790706929784706 us; error 40 us; skew 500 ppm
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:29.792119 22304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:29.793119 22304 webserver.cc:460] Webserver started at http://127.0.98.62:33117/ using document root <none> and password file <none>
I20260929 18:35:29.793663 22304 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:29.793723 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:29.793887 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:29.795846 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "1310f5c21963436a89b550d0eac2e192"
format_stamp: "Formatted at 2026-09-29 18:35:29 on dist-test-slave-fmv8"
server_key: "a600a917373dc5fdd613798dc918234b"
server_key_iv: "31f3d552cc1e56990ee9a8e70f02b92a"
server_key_version: "encryptionkey@0"
I20260929 18:35:29.796409 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "1310f5c21963436a89b550d0eac2e192"
format_stamp: "Formatted at 2026-09-29 18:35:29 on dist-test-slave-fmv8"
server_key: "a600a917373dc5fdd613798dc918234b"
server_key_iv: "31f3d552cc1e56990ee9a8e70f02b92a"
server_key_version: "encryptionkey@0"
I20260929 18:35:29.800010 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:29.802486 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:29.803792 22304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:35:29.803949 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "1310f5c21963436a89b550d0eac2e192"
format_stamp: "Formatted at 2026-09-29 18:35:29 on dist-test-slave-fmv8"
server_key: "a600a917373dc5fdd613798dc918234b"
server_key_iv: "31f3d552cc1e56990ee9a8e70f02b92a"
server_key_version: "encryptionkey@0"
I20260929 18:35:29.804054 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:29.829268 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:29.832687 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:29.832903 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:29.840127 22304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39809
I20260929 18:35:29.840137 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39809 every 8 connection(s)
I20260929 18:35:29.841143 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:29.843894 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:29.847560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22304
I20260929 18:35:29.847673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:29.848004 392 external_mini_cluster.cc:1428] Setting key 8c2a833d1d17efd7fc3953a7e3320961
I20260929 18:35:29.849875 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192: Bootstrap starting.
I20260929 18:35:29.852288 22372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:29.853021 22372 log.cc:824] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:29.855166 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192: No bootstrap required, opened a new log
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:29.859481 22372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } }
I20260929 18:35:29.859742 22372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:29.859814 22372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1310f5c21963436a89b550d0eac2e192, State: Initialized, Role: FOLLOWER
I20260929 18:35:29.860354 22372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } }
I20260929 18:35:29.860541 22372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:29.860667 22372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:29.860793 22372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:29.862054 22372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } }
I20260929 18:35:29.862422 22372 leader_election.cc:302] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1310f5c21963436a89b550d0eac2e192; no voters:
I20260929 18:35:29.862777 22372 leader_election.cc:288] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:29.862872 22377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:29.863113 22377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [term 1 LEADER]: Becoming Leader. State: Replica: 1310f5c21963436a89b550d0eac2e192, State: Running, Role: LEADER
I20260929 18:35:29.863451 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:29.849387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35865 (local address 127.0.98.62:39809)
0929 18:35:29.849827 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:29.849842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:29.849908 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:35:29.850605 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:29.850624 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:29.850975 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:29.851323 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:29.852189 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:29.853303 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:29.854461 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:29.854956 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:29.855087 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:29.858016 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:29.858021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:29.858035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:29.858062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:29.859994 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:29.860487 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:29.860493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:29.860499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:29.860647 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:29.860954 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:29.860961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:29.860963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:29.861497 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:29.861715 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:29.862037 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 18:35:29.862398 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260929 18:35:29.863430 22377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } }
I20260929 18:35:29.864048 22372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:29.865139 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1310f5c21963436a89b550d0eac2e192. Latest consensus state: current_term: 1 leader_uuid: "1310f5c21963436a89b550d0eac2e192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } } }
I20260929 18:35:29.865269 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:29.865630 22378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1310f5c21963436a89b550d0eac2e192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1310f5c21963436a89b550d0eac2e192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39809 } } }
I20260929 18:35:29.865732 22378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:29.867125 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:29.867727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:29.870477 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:29.876389 22386 catalog_manager.cc:1347] Generated new cluster ID: 559d2a0d62dc4150a5f7c5504e0afb90
I20260929 18:35:29.876456 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:29.878381 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:29.869859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34167 (local address 127.0.98.62:39809)
0929 18:35:29.869965 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:29.869971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:29.870173 (+ 202us) server_negotiation.cc:407] Connection header received
0929 18:35:29.870297 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:29.870300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:29.870346 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:29.870432 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:29.871266 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:29.872083 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:29.872948 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:29.873190 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:29.873302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:29.874811 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:29.874816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:29.874819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:29.874844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:29.876666 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:29.877140 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:29.877144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:29.877146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:29.877235 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:29.877518 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:29.877521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:29.877523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:29.877752 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:29.877880 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:29.878101 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:35:29.878221 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:35:29.893801 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:29.894770 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:29.905818 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1310f5c21963436a89b550d0eac2e192: Generated new TSK 0
I20260929 18:35:29.906777 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:29.918505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22304
2026-09-29T18:35:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22403](info): set up 2 sockets
Sep 29 18:35:29 dist-test-slave-fmv8 krb5kdc[22403](info): commencing operation
krb5kdc: starting...
W20260929 18:35:30.960748 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:31 dist-test-slave-fmv8 krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:31.083549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:31.195923 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:31.196283 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:31.196370 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:31.200541 22416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:31.200646 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:31.200696 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:31.200736 22416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:31.200776 22416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:31.206808 22416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:31.207911 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:31.209004 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:31.217783 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:31.217795 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:31.217815 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:31.217891 22416 server_base.cc:1034] running on GCE node
I20260929 18:35:31.218626 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:31.219805 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:31.220996 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790706931220950 us; error 58 us; skew 500 ppm
Sep 29 18:35:31 dist-test-slave-fmv8 krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:31.223690 22416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:31.224692 22416 webserver.cc:460] Webserver started at http://127.0.98.62:34689/ using document root <none> and password file <none>
I20260929 18:35:31.225286 22416 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:31.225394 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:31.225628 22416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:31.227648 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "8200943b59c74b7999c68d567b66399d"
format_stamp: "Formatted at 2026-09-29 18:35:31 on dist-test-slave-fmv8"
server_key: "c78c1b4c3dac8189c718a7c49e7d49e9"
server_key_iv: "f49375daff9f66bf16a6aad535034652"
server_key_version: "encryptionkey@0"
I20260929 18:35:31.228343 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "8200943b59c74b7999c68d567b66399d"
format_stamp: "Formatted at 2026-09-29 18:35:31 on dist-test-slave-fmv8"
server_key: "c78c1b4c3dac8189c718a7c49e7d49e9"
server_key_iv: "f49375daff9f66bf16a6aad535034652"
server_key_version: "encryptionkey@0"
I20260929 18:35:31.232079 22416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:31.234536 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:31.235913 22416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:35:31.236056 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "8200943b59c74b7999c68d567b66399d"
format_stamp: "Formatted at 2026-09-29 18:35:31 on dist-test-slave-fmv8"
server_key: "c78c1b4c3dac8189c718a7c49e7d49e9"
server_key_iv: "f49375daff9f66bf16a6aad535034652"
server_key_version: "encryptionkey@0"
I20260929 18:35:31.236217 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:31.249915 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:31.253262 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:31.253464 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:31.260883 22416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42567
I20260929 18:35:31.260900 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42567 every 8 connection(s)
I20260929 18:35:31.261925 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:31.264688 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:31.270192 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d: Bootstrap starting.
I20260929 18:35:31.270223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22416
I20260929 18:35:31.270399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:31.270699 392 external_mini_cluster.cc:1428] Setting key eda631661786aba3ed328deeb45763c3
I20260929 18:35:31.272995 22484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:31.273869 22484 log.cc:824] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:31.276290 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d: No bootstrap required, opened a new log
I20260929 18:35:31.281023 22484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } }
I20260929 18:35:31.281299 22484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:31.281324 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:31.272117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47427 (local address 127.0.98.62:42567)
0929 18:35:31.272623 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:31.272642 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:31.272684 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:35:31.273560 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:31.273583 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:31.274026 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:31.274401 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:31.274855 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:31.276339 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:31.277328 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:31.278023 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:31.278194 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:31.279113 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:31.279120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:31.279142 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:31.279767 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:31.279834 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:31.279863 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:31.279965 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:31.280094 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:35:31.280354 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":130,"threads_started":1}
I20260929 18:35:31.281360 22484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8200943b59c74b7999c68d567b66399d, State: Initialized, Role: FOLLOWER
I20260929 18:35:31.281838 22484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } }
I20260929 18:35:31.282013 22484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:31.282104 22484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:31.282205 22484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:31.283421 22484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } }
I20260929 18:35:31.283806 22484 leader_election.cc:302] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8200943b59c74b7999c68d567b66399d; no voters:
I20260929 18:35:31.284214 22484 leader_election.cc:288] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:31.284391 22489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:31.284471 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:31.284646 22489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [term 1 LEADER]: Becoming Leader. State: Replica: 8200943b59c74b7999c68d567b66399d, State: Running, Role: LEADER
I20260929 18:35:31.285033 22489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } }
I20260929 18:35:31.285600 22484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:31.286716 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8200943b59c74b7999c68d567b66399d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } } }
I20260929 18:35:31.286834 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:31.287063 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8200943b59c74b7999c68d567b66399d. Latest consensus state: current_term: 1 leader_uuid: "8200943b59c74b7999c68d567b66399d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8200943b59c74b7999c68d567b66399d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42567 } } }
I20260929 18:35:31.287139 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:31.287201 22503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:31.290274 22503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:31.292022 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:31.286732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56659 (local address 127.0.98.62:42567)
0929 18:35:31.286839 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:31.286857 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:31.287819 (+ 962us) server_negotiation.cc:407] Connection header received
0929 18:35:31.287972 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:31.287975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:31.288022 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:31.288104 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:31.288451 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:31.289194 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:31.290274 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:31.290596 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:31.290703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:31.291571 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:31.291575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:31.291578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:31.291591 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:31.291621 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:31.291634 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:31.291715 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:31.291821 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:35:31.291872 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:35:31.296233 22503 catalog_manager.cc:1347] Generated new cluster ID: 9aa58b4d47bf48d5ae1dd978eb58a48b
I20260929 18:35:31.296301 22503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:31.304483 22503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:31.305786 22503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:31.317440 22503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8200943b59c74b7999c68d567b66399d: Generated new TSK 0
I20260929 18:35:31.318230 22503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:31.326812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22416
2026-09-29T18:35:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:31 dist-test-slave-fmv8 krb5kdc[22515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:31 dist-test-slave-fmv8 krb5kdc[22515](info): set up 2 sockets
Sep 29 18:35:31 dist-test-slave-fmv8 krb5kdc[22515](info): commencing operation
krb5kdc: starting...
W20260929 18:35:32.360728 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:32 dist-test-slave-fmv8 krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:32.481438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:32.591894 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:32.592242 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:32.592290 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:32.596247 22528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:32.596302 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:32.596320 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:32.596338 22528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:32.596352 22528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:32.601689 22528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:32.603259 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:32.604862 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:32.613950 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:32.613940 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:32.614164 22528 server_base.cc:1034] running on GCE node
W20260929 18:35:32.613940 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:32.614964 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:32.616088 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:32.617257 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790706932617239 us; error 32 us; skew 500 ppm
Sep 29 18:35:32 dist-test-slave-fmv8 krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:32.620441 22528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:32.621328 22528 webserver.cc:460] Webserver started at http://127.0.98.62:46371/ using document root <none> and password file <none>
I20260929 18:35:32.621860 22528 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:32.621918 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:32.622088 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:32.624089 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "da3a0ee93f284b37b9c385ce4e2861fe"
format_stamp: "Formatted at 2026-09-29 18:35:32 on dist-test-slave-fmv8"
server_key: "c840743807433b5730eb3ad8fdb5034b"
server_key_iv: "7dc9decf00e7d205d505418424bd97ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:32.624662 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "da3a0ee93f284b37b9c385ce4e2861fe"
format_stamp: "Formatted at 2026-09-29 18:35:32 on dist-test-slave-fmv8"
server_key: "c840743807433b5730eb3ad8fdb5034b"
server_key_iv: "7dc9decf00e7d205d505418424bd97ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:32.628641 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:32.631058 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:32.632391 22528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:32.632519 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "da3a0ee93f284b37b9c385ce4e2861fe"
format_stamp: "Formatted at 2026-09-29 18:35:32 on dist-test-slave-fmv8"
server_key: "c840743807433b5730eb3ad8fdb5034b"
server_key_iv: "7dc9decf00e7d205d505418424bd97ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:32.632624 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:32.637248 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:32.640519 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:32.640713 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:32.647687 22528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40029
I20260929 18:35:32.647742 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40029 every 8 connection(s)
I20260929 18:35:32.648675 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:32.651446 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:32.656908 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe: Bootstrap starting.
I20260929 18:35:32.657708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22528
I20260929 18:35:32.657858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:32.658119 392 external_mini_cluster.cc:1428] Setting key e26a5e122d69117d1ac110f2d79f2961
I20260929 18:35:32.659762 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:32.660552 22596 log.cc:824] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:32.662791 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe: No bootstrap required, opened a new log
I20260929 18:35:32.663445 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:32.659363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51925 (local address 127.0.98.62:40029)
0929 18:35:32.659812 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:32.659830 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:32.659871 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:35:32.660608 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:32.660652 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:32.660970 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:32.661245 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:32.662033 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:32.662039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:32.662054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:32.662066 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:32.662131 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:32.662157 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:32.662269 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:32.662357 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:35:32.662503 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260929 18:35:32.666239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:32.666661 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260929 18:35:32.666864 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:32.666934 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3a0ee93f284b37b9c385ce4e2861fe, State: Initialized, Role: FOLLOWER
I20260929 18:35:32.667337 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260929 18:35:32.667488 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:32.667575 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:32.667706 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:32.668918 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260929 18:35:32.669309 22596 leader_election.cc:302] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da3a0ee93f284b37b9c385ce4e2861fe; no voters:
I20260929 18:35:32.669672 22596 leader_election.cc:288] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:32.669727 22606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:32.669957 22606 raft_consensus.cc:686] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [term 1 LEADER]: Becoming Leader. State: Replica: da3a0ee93f284b37b9c385ce4e2861fe, State: Running, Role: LEADER
I20260929 18:35:32.670327 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:32.668419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40993 (local address 127.0.98.62:40029)
0929 18:35:32.668552 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:32.668558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:32.668738 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:35:32.668811 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:32.668813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:32.668853 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:32.668943 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:32.669726 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:32.669729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:32.669732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:35:32.669767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:32.669797 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:35:32.669812 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:32.669912 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:32.670165 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:35:32.670216 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:32.670305 22606 consensus_queue.cc:237] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } }
I20260929 18:35:32.671198 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:32.672429 22608 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3a0ee93f284b37b9c385ce4e2861fe. Latest consensus state: current_term: 1 leader_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } } }
I20260929 18:35:32.672452 22607 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3a0ee93f284b37b9c385ce4e2861fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40029 } } }
I20260929 18:35:32.672550 22608 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:32.672554 22607 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:32.672971 22615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:32.675824 22615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:32.681334 22615 catalog_manager.cc:1347] Generated new cluster ID: d29799109ca7475bac3f1bbb188e0e0e
I20260929 18:35:32.681416 22615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:32.697717 22615 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:32.698060 22615 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:32.702989 22615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da3a0ee93f284b37b9c385ce4e2861fe: Generated new TSK 0
I20260929 18:35:32.703758 22615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:32.707279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22528
2026-09-29T18:35:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1380 ms)
[----------] 18 tests from AuthTokenIssuingTest (25617 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:32 dist-test-slave-fmv8 krb5kdc[22627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:32 dist-test-slave-fmv8 krb5kdc[22627](info): set up 2 sockets
Sep 29 18:35:32 dist-test-slave-fmv8 krb5kdc[22627](info): commencing operation
krb5kdc: starting...
W20260929 18:35:33.752575 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:33 dist-test-slave-fmv8 krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:33.875876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:33.986769 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:33.987155 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:33.987231 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:33.987334 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:33.991518 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:33.991622 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:33.991660 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:33.991720 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:33.991756 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:33.997288 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:33.998412 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:33.999471 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:34.008163 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:34.008180 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:34.008234 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:34.008248 22640 server_base.cc:1034] running on GCE node
I20260929 18:35:34.009012 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:34.010174 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:34.011386 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790706934011368 us; error 40 us; skew 500 ppm
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:34.014182 22640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:34.015280 22640 webserver.cc:460] Webserver started at http://127.0.98.62:37865/ using document root <none> and password file <none>
I20260929 18:35:34.015864 22640 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:34.015930 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:34.016108 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:34.018205 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "64f4442b44016eef060ad690fc934bf1"
server_key_iv: "84f7b65fc734af6029ef61b717c7445e"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.018730 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "64f4442b44016eef060ad690fc934bf1"
server_key_iv: "84f7b65fc734af6029ef61b717c7445e"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.022578 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:35:34.025013 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:34.026294 22640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:34.026408 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "64f4442b44016eef060ad690fc934bf1"
server_key_iv: "84f7b65fc734af6029ef61b717c7445e"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.026496 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:34.055989 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:34.059094 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:34.059268 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:34.066543 22640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46707
I20260929 18:35:34.066874 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46707 every 8 connection(s)
I20260929 18:35:34.067548 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:34.070338 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:34.072739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22640
I20260929 18:35:34.072876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:34.073158 392 external_mini_cluster.cc:1428] Setting key 4ede6e016e2b44c52c20fcbad6b961db
I20260929 18:35:34.075165 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:34.091564 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.078343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.0.98.62:46707)
0929 18:35:34.078617 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.078626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.078662 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:35:34.079868 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.079873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.080061 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.080338 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.081427 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.082384 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.083472 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.083864 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.084040 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.086456 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.086461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.086472 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.086491 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.088381 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.088983 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.088988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.088989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.089100 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.089525 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.089528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.089529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.089941 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.090052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.090518 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.090727 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260929 18:35:34.091571 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.077603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46707 (local address 127.0.0.1:49496)
0929 18:35:34.078186 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.078251 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.079514 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.080365 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.080373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.080864 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.081260 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.081280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.082522 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.082525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.083338 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.083350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.084060 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.084067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.084225 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.084895 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.084911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.086304 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.088494 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.088507 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.088519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.088856 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.089229 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.089235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.089240 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.089401 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.090066 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.090074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.090251 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.090444 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.090847 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":77,"thread_start_us":113,"threads_started":1}
I20260929 18:35:34.094405 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:34.095191 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.094883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51634)
0929 18:35:34.095046 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.095132 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:35:34.095384 22658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
I20260929 18:35:34.096388 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.096068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51650)
0929 18:35:34.096276 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.096333 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 18:35:34.096544 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42089: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 18:35:34.100880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42089
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:34.145862 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } attempt: 1
I20260929 18:35:34.146795 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.146480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51654)
0929 18:35:34.146641 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.146702 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 18:35:34.147569 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.147378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51666)
0929 18:35:34.147478 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.147521 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:35:34.147799 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42089: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
W20260929 18:35:34.211040 22715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:34.211306 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:34.211329 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:34.211419 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:34.215458 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:34.215509 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:34.215528 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:34.215545 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:34.215560 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:34.216008 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } attempt: 2
I20260929 18:35:34.216886 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.216594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51672)
0929 18:35:34.216745 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.216813 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 18:35:34.217602 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.217411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51682)
0929 18:35:34.217511 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.217554 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:35:34.217749 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42089: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42089: connect: Connection refused (error 111)
I20260929 18:35:34.221297 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:34.222285 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:34.223210 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:34.232070 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:34.232076 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:34.232218 22715 server_base.cc:1034] running on GCE node
W20260929 18:35:34.232399 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:34.232924 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:34.234010 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:34.235250 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790706934235240 us; error 27 us; skew 500 ppm
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:34.238071 22715 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 18:35:34.238972 22715 webserver.cc:460] Webserver started at http://127.0.98.61:38843/ using document root <none> and password file <none>
I20260929 18:35:34.239524 22715 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:34.239611 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:34.239835 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:34.241894 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data/instance:
uuid: "cd5ae453c66d42cbb291660cc4dcee8b"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "020b886573ff5a864c7cd6ca9230e3de"
server_key_iv: "5d3eacecfad8b70b652755e9b33f1ebf"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.242457 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal/instance:
uuid: "cd5ae453c66d42cbb291660cc4dcee8b"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "020b886573ff5a864c7cd6ca9230e3de"
server_key_iv: "5d3eacecfad8b70b652755e9b33f1ebf"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.246165 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:34.248720 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:34.249925 22715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:34.250069 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "cd5ae453c66d42cbb291660cc4dcee8b"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "020b886573ff5a864c7cd6ca9230e3de"
server_key_iv: "5d3eacecfad8b70b652755e9b33f1ebf"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.250185 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:34.261472 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:34.264827 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:34.265039 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:34.273130 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42089
I20260929 18:35:34.273419 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42089 every 8 connection(s)
I20260929 18:35:34.274149 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:35:34.277026 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:34.277374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22715
I20260929 18:35:34.277514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-1/wal/instance
I20260929 18:35:34.277827 392 external_mini_cluster.cc:1428] Setting key 2821a24f59d570ac6656fce0b81ac9f4
I20260929 18:35:34.280963 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:34.295459 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.282932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.0.98.62:46707)
0929 18:35:34.283120 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.283128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.283908 (+ 780us) server_negotiation.cc:407] Connection header received
0929 18:35:34.284926 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.284932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.285006 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.285123 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.286317 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.286927 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.288325 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.288646 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.288731 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.291769 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.291772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.291774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.291800 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.293343 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.293873 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.293877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.293879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.293968 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.294315 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.294320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.294322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.294511 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.294621 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.295162 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.295291 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:35:34.296437 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.283279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46707 (local address 127.0.0.1:49510)
0929 18:35:34.283736 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.283786 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.284710 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.285269 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.285277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.285691 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.286162 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.286175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.287072 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.287075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.288142 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.288183 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.288859 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.288867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.289112 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.289811 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.289840 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.291580 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.293472 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.293478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.293493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.293762 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.294069 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.294073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.294075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.294219 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.294621 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.294629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.294910 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.295107 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.295512 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260929 18:35:34.298005 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 18:35:34.304901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40479
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:34.307755 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.298560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51698)
0929 18:35:34.298682 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.298697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.298798 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.299440 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.299444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.299612 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.299711 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.299716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.300806 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.300809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.301331 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.301339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.301995 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.301999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.302085 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.302645 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.302657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.304104 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.306205 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.306211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.306215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.306406 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.306702 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.306706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.306708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.306788 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.307234 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.307236 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.307321 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.307451 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.307581 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:35:34.307802 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.298745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.98.61:42089)
0929 18:35:34.299085 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.299094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.299111 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:34.299142 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.299146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.299296 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.299418 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.299827 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.300703 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.301444 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.301879 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.301999 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.304291 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.304299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.304312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.304332 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.306094 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.306521 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.306534 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.306536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.306608 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.306884 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.306887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.306889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.307124 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.307268 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.307543 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.307667 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":170,"threads_started":1}
I20260929 18:35:34.308914 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } attempt: 3
I20260929 18:35:34.310242 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:34.311151 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.310847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37632)
0929 18:35:34.310999 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.311078 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 18:35:34.311371 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
I20260929 18:35:34.312767 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.312556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37634)
0929 18:35:34.312659 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.312705 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:35:34.312952 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:35:34.319156 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.309418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51700)
0929 18:35:34.309699 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.309719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.309829 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.310273 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.310279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.310496 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.310668 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.310679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.311666 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.311673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.313245 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.313260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.313727 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.313733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.313874 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.314347 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.314363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.315755 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.317664 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.317666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.317668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.317863 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.318136 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.318138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.318139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.318227 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.318698 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.318703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.318816 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.318920 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.319037 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 18:35:34.319247 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.309518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.0.98.61:42089)
0929 18:35:34.309628 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.309632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.309788 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:35:34.309935 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.309937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.309973 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.310216 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.310803 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.311530 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.313372 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.313607 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.313691 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.315894 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.315899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.315902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.315931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.317553 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.317958 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.317962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.317964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.318046 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.318309 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.318312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.318314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.318524 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.318618 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.318989 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.319103 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:35:34.320428 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:34.321192 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.320942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37642)
0929 18:35:34.321063 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.321120 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:35:34.321924 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.321671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37648)
0929 18:35:34.321822 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.321870 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 18:35:34.322100 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
I20260929 18:35:34.353324 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 1
I20260929 18:35:34.354249 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.353898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37650)
0929 18:35:34.354077 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.354152 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:35:34.355013 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.354783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37660)
0929 18:35:34.354897 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.354952 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:35:34.355159 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
I20260929 18:35:34.362217 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 1
I20260929 18:35:34.363130 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.362739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37666)
0929 18:35:34.362873 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.362936 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:35:34.363871 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.363653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37676)
0929 18:35:34.363755 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.363809 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:35:34.364001 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
W20260929 18:35:34.423027 22789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:34.423386 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:34.423460 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:34.423537 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:34.427712 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:34.427791 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:34.427829 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:34.427889 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:34.427928 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:34.430364 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 2
I20260929 18:35:34.431358 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.431043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37686)
0929 18:35:34.431199 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.431268 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:35:34.432132 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.431916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37700)
0929 18:35:34.432023 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.432072 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:35:34.432271 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
I20260929 18:35:34.432215 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 2
I20260929 18:35:34.432865 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.432600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37704)
0929 18:35:34.432728 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.432785 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:35:34.433591 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.433380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37714)
0929 18:35:34.433489 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.433536 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:35:34.433786 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40479: connect: Connection refused (error 111)
I20260929 18:35:34.434221 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46707,127.0.98.61:42089,127.0.98.60:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:34.435276 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:34.436357 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:35:34.444967 22789 server_base.cc:1034] running on GCE node
W20260929 18:35:34.444867 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:34.444886 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:34.444890 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:34.445744 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:34.446797 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:34.447974 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790706934447944 us; error 40 us; skew 500 ppm
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:34.450877 22789 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 18:35:34.451834 22789 webserver.cc:460] Webserver started at http://127.0.98.60:44433/ using document root <none> and password file <none>
I20260929 18:35:34.452450 22789 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:34.452520 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:34.452747 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:34.454931 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data/instance:
uuid: "b471721958754f269be1e50f5a8ac8b2"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "806a0f029ef794f9c79ecf964e6a76fa"
server_key_iv: "b52093499b1f9681af7038d20143533a"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.455531 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal/instance:
uuid: "b471721958754f269be1e50f5a8ac8b2"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "806a0f029ef794f9c79ecf964e6a76fa"
server_key_iv: "b52093499b1f9681af7038d20143533a"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.459175 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:35:34.461988 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:34.463366 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 18:35:34.463541 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "b471721958754f269be1e50f5a8ac8b2"
format_stamp: "Formatted at 2026-09-29 18:35:34 on dist-test-slave-fmv8"
server_key: "806a0f029ef794f9c79ecf964e6a76fa"
server_key_iv: "b52093499b1f9681af7038d20143533a"
server_key_version: "encryptionkey@0"
I20260929 18:35:34.463656 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:34.491204 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:34.494299 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:34.494518 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:34.501711 22789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40479
I20260929 18:35:34.502072 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40479 every 8 connection(s)
I20260929 18:35:34.502714 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:35:34.505548 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:34.509230 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:34.512063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 22789
I20260929 18:35:34.512204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706655775522-392-0/minicluster-data/master-2/wal/instance
I20260929 18:35:34.512508 392 external_mini_cluster.cc:1428] Setting key aa402528b4ddbed3edb4e5bc64405cd0
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:34.524971 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.511096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.62:46707)
0929 18:35:34.511252 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.511257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.512349 (+ 1092us) server_negotiation.cc:407] Connection header received
0929 18:35:34.513515 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.513518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.513577 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.513691 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.514892 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.515706 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.517278 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.517616 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.517735 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.520721 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.520725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.520728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.520777 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.522365 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.522924 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.522932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.522934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.523020 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.523655 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.523659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.523661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.523988 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.524115 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.524712 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.524820 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:35:34.524972 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 3
I20260929 18:35:34.526305 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.513175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.0.98.60:40479)
0929 18:35:34.513553 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.513565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.513591 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:35:34.513817 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.513821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.513982 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.514118 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.515011 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.516132 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.517426 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.517991 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.518076 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.521466 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.521472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.521483 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.521504 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.523159 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.523617 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.523628 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.523629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.523702 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.523979 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.523984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.523986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.524418 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.524647 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.524833 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.525361 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":290,"thread_start_us":101,"threads_started":1}
I20260929 18:35:34.526903 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.511512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46707 (local address 127.0.0.1:49516)
0929 18:35:34.512128 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.512215 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.513249 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.513828 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.513835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.514264 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.514729 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.514750 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.515849 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.515852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.517118 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.517129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.517779 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.517782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.517978 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.518728 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.518748 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.520536 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.522506 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.522517 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.522528 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.522810 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.523349 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.523354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.523384 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.523542 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.524112 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.524120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.524431 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.525125 (+ 694us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.526732 (+ 1607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":1411,"thread_start_us":123,"threads_started":1}
I20260929 18:35:34.528527 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:34.531396 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } attempt: 3
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:34.536273 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.525441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37746)
0929 18:35:34.525570 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.525586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.525690 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.526004 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.526008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.526171 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.526257 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.526263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.527077 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.527081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.527818 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.527829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.528412 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.528415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.528501 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.529220 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.529233 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.530771 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.534700 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.534703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.534705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.534911 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.535206 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.535209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.535211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.535292 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.535693 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.535696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.535795 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.535908 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.536040 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:35:34.536877 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.525488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.0.98.60:40479)
0929 18:35:34.525794 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.525809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.525819 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:35:34.525845 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.525847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.525886 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.525980 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.526369 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.526966 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.527939 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.528288 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.528402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.530908 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.530911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.530914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.530937 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.534606 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.535013 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.535018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.535020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.535106 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.535413 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.535416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.535418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.535593 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.535702 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.536605 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.536716 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:34.539825 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.529062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42089 (local address 127.0.0.1:51708)
0929 18:35:34.529855 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.529872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.529963 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.530207 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.530210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.530379 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.530487 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.530493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.531451 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.531454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.532307 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.532317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.532946 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.532949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.533057 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.533648 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.533666 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.535684 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.538195 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.538200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.538203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.538453 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.538780 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.538783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.538785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.538866 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.539275 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.539278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.539372 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.539479 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.539684 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721}
I20260929 18:35:34.539825 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.529146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.0.98.61:42089)
0929 18:35:34.529255 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.529260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.529933 (+ 673us) server_negotiation.cc:407] Connection header received
0929 18:35:34.530063 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.530068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.530110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.530193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.530629 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.531323 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.532427 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.532741 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.532810 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.536372 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.536377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.536380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.536409 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.538061 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.538568 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.538572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.538574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.538662 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.538961 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.538966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.538968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.539172 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.539284 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.539556 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.539683 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:34.540475 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:34.541455 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.531758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37762)
0929 18:35:34.531905 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.531917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.531981 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.532373 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.532379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.532561 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.532665 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.532671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.534325 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.534332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.535214 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.535225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.535919 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.535922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.536010 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.536449 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.536462 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.537828 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.539899 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.539902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.539904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.540169 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.540498 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.540501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.540504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.540601 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.541013 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.541016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.541103 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.541180 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.541299 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 18:35:34.541693 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: Bootstrap starting.
I20260929 18:35:34.542071 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.531833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.98.60:40479)
0929 18:35:34.532110 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.532115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.532125 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:35:34.532208 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.532211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.532246 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.532774 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.532854 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.534191 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.535436 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.535800 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.535902 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.538002 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.538005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.538008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.538033 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.539778 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.540296 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.540299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.540301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.540392 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.540717 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.540720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.540721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.540898 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.541638 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.541762 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.541905 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260929 18:35:34.545521 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:34.545773 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Bootstrap starting.
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:34.546404 22783 log.cc:824] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:34.548712 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: No bootstrap required, opened a new log
I20260929 18:35:34.548800 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:34.549726 22708 log.cc:824] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:34.549891 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.529211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.98.62:46707)
0929 18:35:34.529313 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.529317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.529480 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:35:34.529548 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.529551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.529598 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.529684 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.530570 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.531267 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.532333 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.532649 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.532762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.536309 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.536312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.536314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.536364 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.547536 (+ 11172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.548103 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.548106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.548108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.548221 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.548542 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.548545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.548547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.548768 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.548947 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.549631 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.549744 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:35:34.550132 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.540964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40479 (local address 127.0.0.1:37770)
0929 18:35:34.541148 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:34.541158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:34.541247 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:34.541448 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:34.541451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:34.541635 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:34.541726 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.541735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.543356 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.543359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.543896 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:34.543903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.544511 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.544515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:34.544596 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.545087 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:34.545101 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:34.546476 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:34.548579 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.548581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.548583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.548826 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.549131 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:34.549133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:34.549135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:34.549204 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:34.549638 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:34.549641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:34.549720 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 18:35:34.549824 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:35:34.549961 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 18:35:34.550560 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.540990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.0.98.60:40479)
0929 18:35:34.541092 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.541095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.541212 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:35:34.541316 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.541320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.541352 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.541441 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.541952 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.543225 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.544014 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.544384 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.544486 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.546614 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.546618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.546620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.546642 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.548427 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.548947 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.548950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.548951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.549034 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.549288 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.549290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.549292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.549469 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.549600 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.550306 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.550414 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:35:34.551792 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: No bootstrap required, opened a new log
I20260929 18:35:34.553871 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: Bootstrap starting.
I20260929 18:35:34.554009 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.554242 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5ae453c66d42cbb291660cc4dcee8b, State: Initialized, Role: FOLLOWER
I20260929 18:35:34.554566 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.555009 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.555240 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5f80c8e84f4ab6b09a7fa5a1c9b781, State: Initialized, Role: FOLLOWER
I20260929 18:35:34.555145 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.555440 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.555560 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.556005 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:34.556027 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.556291 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.556609 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:34.556893 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:34.557503 22858 log.cc:824] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:34.560761 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: No bootstrap required, opened a new log
W20260929 18:35:34.560822 22661 tablet.cc:2369] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:34.561324 22884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:34.561461 22884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:34.563715 22715 master_runner.cc:186] Error getting master registration for 127.0.98.60:40479: OK, instance_id { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" instance_seqno: 1790706934499719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:35:34.567176 22715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:34.564630 22640 master_runner.cc:186] Error getting master registration for 127.0.98.60:40479: OK, instance_id { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" instance_seqno: 1790706934499719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:35:34.569005 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 18:35:34 dist-test-slave-fmv8 krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:35:34.569417 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.569700 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b471721958754f269be1e50f5a8ac8b2, State: Initialized, Role: FOLLOWER
I20260929 18:35:34.570091 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.570675 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.570868 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.571743 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:34.574236 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.552279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.0.98.61:42089)
0929 18:35:34.552398 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.552402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.558139 (+ 5737us) server_negotiation.cc:407] Connection header received
0929 18:35:34.558356 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.558362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.558420 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.558493 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.563058 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.563805 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.566606 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.566942 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.567019 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.570282 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.570285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.570289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.570317 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.572049 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.572600 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.572603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.572605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.572703 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.573067 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.573073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.573078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.573336 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.573512 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.573899 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.574057 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:34.578135 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:35:34.577965 22890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:34.579380 22890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:34.579843 22682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49520
I20260929 18:35:34.579986 22789 master_runner.cc:263] No leader master found from master b471721958754f269be1e50f5a8ac8b2
I20260929 18:35:34.580052 22789 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:34.580066 22682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:35:34.580210 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:34.581282 22682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.582206 22682 leader_election.cc:288] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [CANDIDATE]: Term 1 election: Requested vote from peers cd5ae453c66d42cbb291660cc4dcee8b (127.0.98.61:42089), b471721958754f269be1e50f5a8ac8b2 (127.0.98.60:40479)
I20260929 18:35:34.582612 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd5ae453c66d42cbb291660cc4dcee8b"
I20260929 18:35:34.582864 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:34.582933 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b471721958754f269be1e50f5a8ac8b2"
I20260929 18:35:34.583138 22832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:34.584277 22757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f80c8e84f4ab6b09a7fa5a1c9b781 in term 1.
I20260929 18:35:34.584451 22832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f5f80c8e84f4ab6b09a7fa5a1c9b781 in term 1.
I20260929 18:35:34.584684 22658 leader_election.cc:302] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5f80c8e84f4ab6b09a7fa5a1c9b781, cd5ae453c66d42cbb291660cc4dcee8b; no voters:
I20260929 18:35:34.584959 22867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:34.585158 22867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [term 1 LEADER]: Becoming Leader. State: Replica: 3f5f80c8e84f4ab6b09a7fa5a1c9b781, State: Running, Role: LEADER
I20260929 18:35:34.585469 22867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } }
I20260929 18:35:34.587585 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f5f80c8e84f4ab6b09a7fa5a1c9b781. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.587780 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:34.588215 22912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 18:35:34.588361 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:34.588444 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:34.589036 22912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:34.593735 22832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [term 1 FOLLOWER]: Refusing update from remote peer 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:34.594439 22910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:34.599030 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f5f80c8e84f4ab6b09a7fa5a1c9b781. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.599145 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.600296 22757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [term 1 FOLLOWER]: Refusing update from remote peer 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:34.600900 22910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:34.610793 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f5f80c8e84f4ab6b09a7fa5a1c9b781. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.610946 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.618968 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.619112 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:34.619393 22913 mvcc.cc:205] Tried to move back new op lower bound from 7334735604088311808 to 7334735604064886784. Current Snapshot: MvccSnapshot[applied={T|T < 7334735604088311808}]
I20260929 18:35:34.619599 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.619737 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.620589 22919 mvcc.cc:205] Tried to move back new op lower bound from 7334735604088311808 to 7334735604064886784. Current Snapshot: MvccSnapshot[applied={T|T < 7334735604088311808}]
I20260929 18:35:34.620826 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.620962 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:34.621524 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.592610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.98.60:40479)
0929 18:35:34.592741 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.592745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.595010 (+ 2265us) server_negotiation.cc:407] Connection header received
0929 18:35:34.596352 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.596355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.596402 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.596459 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.599392 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.600055 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.602839 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.603386 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.603540 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.613691 (+ 10151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.613695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.613698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.613731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.617124 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.617862 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.617866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.617867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.617973 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.619978 (+ 2005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.619982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.619985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.620267 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.620406 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.621263 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.621370 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:34.621524 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.593809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.0.98.61:42089)
0929 18:35:34.593919 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.593924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.601299 (+ 7375us) server_negotiation.cc:407] Connection header received
0929 18:35:34.601657 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.601662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.601754 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.601874 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.606579 (+ 4705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.607379 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.608428 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.608770 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.608845 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.610735 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.610738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.610741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.610769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.612786 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.613388 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.613392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.613394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.613487 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.613791 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.613794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.613796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.617336 (+ 3540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.617449 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.621238 (+ 3789us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.621370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:35:34.622116 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5f80c8e84f4ab6b09a7fa5a1c9b781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46707 } } peers { permanent_uuid: "cd5ae453c66d42cbb291660cc4dcee8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42089 } } peers { permanent_uuid: "b471721958754f269be1e50f5a8ac8b2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40479 } } }
I20260929 18:35:34.622248 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:34.623450 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:34.593045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.0.98.62:46707)
0929 18:35:34.593151 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.593157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.594681 (+ 1524us) server_negotiation.cc:407] Connection header received
0929 18:35:34.595085 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.595088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.595142 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.595210 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.597350 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.598146 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.603908 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:34.604542 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:34.604667 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:34.614097 (+ 9430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:34.614104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:34.614107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:34.614141 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:34.615859 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.620779 (+ 4920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.620786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.620790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.620886 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:34.622314 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:34.622319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:34.622320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:34.622596 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:34.622717 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:34.623148 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 18:35:34.623272 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:35:34.624461 22917 mvcc.cc:205] Tried to move back new op lower bound from 7334735604088311808 to 7334735604064886784. Current Snapshot: MvccSnapshot[applied={T|T < 7334735604088311808}]
I20260929 18:35:34.626102 22912 catalog_manager.cc:1347] Generated new cluster ID: 4264c25cd5cd4bb7a8ad5ab84588fa48
I20260929 18:35:34.626169 22912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:34.650051 22912 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:34.651445 22912 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:34.665863 22912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f5f80c8e84f4ab6b09a7fa5a1c9b781: Generated new TSK 0
I20260929 18:35:34.666939 22912 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:35:34.700595 22863 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.697379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.98.60:40479)
0929 18:35:34.697508 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.697512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.699445 (+ 1933us) server_negotiation.cc:407] Connection header received
0929 18:35:34.699663 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.699668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.699718 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.699772 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.700441 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37798: BlockingRecv error: recv got EOF from 127.0.0.1:37798 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 18:35:34.702090 22712 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.698194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.98.62:46707)
0929 18:35:34.698304 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.698309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.700970 (+ 2661us) server_negotiation.cc:407] Connection header received
0929 18:35:34.701297 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.701301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.701353 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.701405 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.701974 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49538: BlockingRecv error: recv got EOF from 127.0.0.1:49538 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:35:34.703456 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.698839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.0.98.61:42089)
0929 18:35:34.698954 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.698959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.702464 (+ 3505us) server_negotiation.cc:407] Connection header received
0929 18:35:34.702706 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.702710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.702764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.702816 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.703337 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51726: BlockingRecv error: recv got EOF from 127.0.0.1:51726 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:35:34.708289 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.706644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.98.61:42089)
0929 18:35:34.706765 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.706769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.707252 (+ 483us) server_negotiation.cc:407] Connection header received
0929 18:35:34.707476 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.707479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.707554 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.707619 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.708130 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51736: BlockingRecv error: recv got EOF from 127.0.0.1:51736 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 18:35:34.713717 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.712001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.0.98.61:42089)
0929 18:35:34.712107 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.712111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.712805 (+ 694us) server_negotiation.cc:407] Connection header received
0929 18:35:34.712993 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.712996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.713045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.713096 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.713608 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51742: BlockingRecv error: recv got EOF from 127.0.0.1:51742 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:35:34.719141 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.717659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.98.61:42089)
0929 18:35:34.717761 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.717765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.718268 (+ 503us) server_negotiation.cc:407] Connection header received
0929 18:35:34.718449 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.718452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.718501 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.718551 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.719035 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51750: BlockingRecv error: recv got EOF from 127.0.0.1:51750 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:35:34.726984 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.725257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.98.61:42089)
0929 18:35:34.725399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.725403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.725978 (+ 575us) server_negotiation.cc:407] Connection header received
0929 18:35:34.726182 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.726185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.726256 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.726310 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.726870 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51766: BlockingRecv error: recv got EOF from 127.0.0.1:51766 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 18:35:34.739125 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.737428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.0.98.61:42089)
0929 18:35:34.737560 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.737564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.738125 (+ 561us) server_negotiation.cc:407] Connection header received
0929 18:35:34.738347 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.738350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.738402 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.738454 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.739014 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51770: BlockingRecv error: recv got EOF from 127.0.0.1:51770 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:35:34.759786 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.757905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.98.61:42089)
0929 18:35:34.758052 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.758059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.758710 (+ 651us) server_negotiation.cc:407] Connection header received
0929 18:35:34.758933 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.758937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.758997 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.759054 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.759661 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51778: BlockingRecv error: recv got EOF from 127.0.0.1:51778 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 18:35:34.795972 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.794239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.0.98.61:42089)
0929 18:35:34.794364 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.794368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.794962 (+ 594us) server_negotiation.cc:407] Connection header received
0929 18:35:34.795182 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.795186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.795239 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.795292 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.795847 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51786: BlockingRecv error: recv got EOF from 127.0.0.1:51786 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:35:34.863801 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.862468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.0.98.61:42089)
0929 18:35:34.862775 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.862782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.862862 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:35:34.862993 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.862996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.863049 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.863173 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.863666 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51796: BlockingRecv error: recv got EOF from 127.0.0.1:51796 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 18:35:34.995330 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:34.994096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.0.98.61:42089)
0929 18:35:34.994262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:34.994267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:34.994396 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:35:34.994520 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:34.994522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:34.994571 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:34.994644 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:34.995168 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51804: BlockingRecv error: recv got EOF from 127.0.0.1:51804 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 18:35:35.254894 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:35.253653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.98.61:42089)
0929 18:35:35.253830 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.253836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.253853 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:35.254024 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.254027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.254085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.254180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:35.254714 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51808: BlockingRecv error: recv got EOF from 127.0.0.1:51808 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:35:35.562841 22884 catalog_manager.cc:1259] Loaded cluster ID: 4264c25cd5cd4bb7a8ad5ab84588fa48
I20260929 18:35:35.562937 22884 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: loading cluster ID for follower catalog manager: success
I20260929 18:35:35.564792 22884 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: acquiring CA information for follower catalog manager: success
I20260929 18:35:35.565901 22884 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd5ae453c66d42cbb291660cc4dcee8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:35.589782 22904 catalog_manager.cc:1259] Loaded cluster ID: 4264c25cd5cd4bb7a8ad5ab84588fa48
I20260929 18:35:35.589872 22904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: loading cluster ID for follower catalog manager: success
I20260929 18:35:35.591454 22904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: acquiring CA information for follower catalog manager: success
I20260929 18:35:35.592592 22904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b471721958754f269be1e50f5a8ac8b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:35.773248 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:35.769185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.0.98.61:42089)
0929 18:35:35.769462 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.769466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.769483 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:35.769588 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.769590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.769735 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.769834 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:35.770072 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.770610 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.771438 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.771723 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.771812 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:35.772122 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:35.772654 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:35.772810 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:35.772879 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:35:35.772936 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260929 18:35:35.779835 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:35.776298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.0.98.60:40479)
0929 18:35:35.776558 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.776562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.776575 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:35.776693 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.776696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.776807 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.776906 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:35.777077 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.777589 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.778401 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.778628 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.778715 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:35.778954 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:35.779375 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:35.779466 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:35.779584 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 18:35:35.779639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260929 18:35:35.786916 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:35.783151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.61:42089)
0929 18:35:35.783272 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.783275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.783712 (+ 437us) server_negotiation.cc:407] Connection header received
0929 18:35:35.783892 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.783896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.783950 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.784018 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:35.784265 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.784968 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.785862 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.786167 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.786263 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:35.786472 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:35.786554 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:35.786625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:35.786735 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:35:35.786783 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:35:35.787716 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:35.782791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.0.98.62:46707)
0929 18:35:35.783053 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.783058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.783162 (+ 104us) server_negotiation.cc:407] Connection header received
0929 18:35:35.783411 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.783415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.783579 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.783709 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:35.784044 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.784698 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.786068 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.786422 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.786522 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:35.786866 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:35.787214 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:35.787281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:35.787458 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:35:35.787526 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260929 18:35:35.787907 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:35.782929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.98.60:40479)
0929 18:35:35.783192 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:35.783196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:35.783415 (+ 219us) server_negotiation.cc:407] Connection header received
0929 18:35:35.783533 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:35.783535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:35.783572 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:35.783658 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:35.783910 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.784709 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.786602 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:35.786921 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:35.787322 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:35.787387 (+ 65us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:35.787478 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:35.787581 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:35.787734 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 18:35:35.787795 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260929 18:35:35.794778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22640
I20260929 18:35:35.801554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22715
I20260929 18:35:35.807227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 22789
2026-09-29T18:35:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3102 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:35 dist-test-slave-fmv8 krb5kdc[23006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:35 dist-test-slave-fmv8 krb5kdc[23006](info): set up 2 sockets
Sep 29 18:35:35 dist-test-slave-fmv8 krb5kdc[23006](info): commencing operation
krb5kdc: starting...
W20260929 18:35:36.940723 392 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/test-user.keytab.
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 18:35:37.060431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:37.169734 23019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:37.170050 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:37.170122 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:37.170205 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:37.174150 23019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:37.174253 23019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:37.174292 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:37.174328 23019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:37.174357 23019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:37.179658 23019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:37.180784 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:37.181802 23019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:37.190274 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:37.190469 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:37.190428 23019 server_base.cc:1034] running on GCE node
W20260929 18:35:37.190639 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:37.191207 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:37.192399 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:37.193626 23019 hybrid_clock.cc:648] HybridClock initialized: now 1790706937193607 us; error 30 us; skew 500 ppm
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:37.196511 23019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 18:35:37.197453 23019 webserver.cc:460] Webserver started at http://127.0.98.62:42969/ using document root <none> and password file <none>
I20260929 18:35:37.198014 23019 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:37.198079 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:37.198310 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:37.200407 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data/instance:
uuid: "8a3b87fc85a54521aeae9a56c7ce341f"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "13fbe07094a1ce8da9567ab47b3cf586"
server_key_iv: "b0c78f408130cfcf07bae4bd315dc2ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.200978 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a3b87fc85a54521aeae9a56c7ce341f"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "13fbe07094a1ce8da9567ab47b3cf586"
server_key_iv: "b0c78f408130cfcf07bae4bd315dc2ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.204630 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:37.206907 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:37.208050 23019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:35:37.208217 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal
uuid: "8a3b87fc85a54521aeae9a56c7ce341f"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "13fbe07094a1ce8da9567ab47b3cf586"
server_key_iv: "b0c78f408130cfcf07bae4bd315dc2ba"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.208362 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:37.219579 23019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:37.222621 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:37.222836 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:37.229947 23019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32877
I20260929 18:35:37.230227 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32877 every 8 connection(s)
I20260929 18:35:37.230926 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:37.233809 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:37.236389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 23019
I20260929 18:35:37.236527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-0/wal/instance
I20260929 18:35:37.236827 392 external_mini_cluster.cc:1428] Setting key 39d1ca5abe8be4a7837c509e5116dfac
I20260929 18:35:37.238046 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 18:35:37.254195 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.240477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32877 (local address 127.0.0.1:44294)
0929 18:35:37.241018 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.241075 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.242402 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.243162 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.243169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.243669 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.244060 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.244082 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.245344 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.245350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.246155 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.246164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.246926 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.246929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.247032 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.247581 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.247606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.249167 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.251113 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.251122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.251132 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.251356 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.251681 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.251684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.251686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.251813 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.252469 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.252475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.252614 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.252786 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.253196 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":110,"thread_start_us":119,"threads_started":1}
I20260929 18:35:37.254195 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.240477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.0.98.62:32877)
0929 18:35:37.241001 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.241022 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.241204 (+ 182us) server_negotiation.cc:407] Connection header received
0929 18:35:37.242750 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.242755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.242939 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.243087 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.244290 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.245214 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.246268 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.246629 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.246765 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.249329 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.249334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.249344 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.249364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.250985 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.251471 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.251483 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.251485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.251589 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.251909 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.251912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.251914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.252350 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.252483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.252855 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.253148 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":48000,"thread_start_us":159,"threads_started":1}
I20260929 18:35:37.256606 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.257445 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.257131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56172)
0929 18:35:37.257307 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.257383 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:35:37.257704 23035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
I20260929 18:35:37.258550 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.258358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56182)
0929 18:35:37.258453 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.258501 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:35:37.258809 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 18:35:37.264530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33177
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:37.308070 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 1
I20260929 18:35:37.309057 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.308748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56194)
0929 18:35:37.308922 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.308982 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 18:35:37.309779 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.309560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56208)
0929 18:35:37.309670 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.309721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:35:37.309918 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
W20260929 18:35:37.376258 23094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:37.376591 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:37.376720 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:37.376804 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:35:37.378082 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 2
I20260929 18:35:37.378921 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.378622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56224)
0929 18:35:37.378772 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.378835 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 18:35:37.379691 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.379474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56240)
0929 18:35:37.379579 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.379627 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:35:37.379868 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33177: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33177: connect: Connection refused (error 111)
W20260929 18:35:37.381567 23094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:37.381618 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:37.381637 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:37.381654 23094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:37.381668 23094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:37.386869 23094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:37.387821 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:37.388811 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:35:37.397346 23094 server_base.cc:1034] running on GCE node
W20260929 18:35:37.397305 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:37.397313 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:37.397290 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:37.398193 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:37.399230 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:37.400422 23094 hybrid_clock.cc:648] HybridClock initialized: now 1790706937400394 us; error 39 us; skew 500 ppm
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:37.403221 23094 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 18:35:37.404255 23094 webserver.cc:460] Webserver started at http://127.0.98.61:35157/ using document root <none> and password file <none>
I20260929 18:35:37.404804 23094 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:37.404893 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:37.405122 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:37.407146 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data/instance:
uuid: "2a6a7299c06c43b48c6674f8b570e935"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "879adca6ec0da220d3873a52c21d466d"
server_key_iv: "34ac5574d3c0476e29f3b470faa99bbb"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.407698 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal/instance:
uuid: "2a6a7299c06c43b48c6674f8b570e935"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "879adca6ec0da220d3873a52c21d466d"
server_key_iv: "34ac5574d3c0476e29f3b470faa99bbb"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.411286 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:35:37.413923 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:37.415094 23094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:37.415261 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal
uuid: "2a6a7299c06c43b48c6674f8b570e935"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "879adca6ec0da220d3873a52c21d466d"
server_key_iv: "34ac5574d3c0476e29f3b470faa99bbb"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.415411 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:37.442019 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:37.445299 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:37.445525 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:37.453008 23094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33177
I20260929 18:35:37.453293 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33177 every 8 connection(s)
I20260929 18:35:37.454130 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/data/info.pb
I20260929 18:35:37.456934 23162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:37.460695 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.461062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 23094
I20260929 18:35:37.461187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-1/wal/instance
I20260929 18:35:37.461508 392 external_mini_cluster.cc:1428] Setting key adb0f68cc627880af9ad1078e8376c47
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 18:35:37.471038 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:35:37.478163 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.463009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.98.62:32877)
0929 18:35:37.463143 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.463148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.464180 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 18:35:37.465128 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.465133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.465204 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.465318 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.466446 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.467365 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.468574 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.469028 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.469150 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.472952 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.472963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.472966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.472989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.474891 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.475403 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.475407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.475408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.475489 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.475856 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.475859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.475861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.476109 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.476236 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.477781 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.477929 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":54}
I20260929 18:35:37.478170 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.463483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32877 (local address 127.0.0.1:44296)
0929 18:35:37.463989 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.464042 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.464903 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.465482 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.465489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.465841 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.466285 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.466297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.467505 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.467507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.468454 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.468463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.469238 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.469242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.469495 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.470070 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.470086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.472811 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.475006 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.475012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.475025 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.475294 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.475580 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.475585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.475587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.475729 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.476236 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.476243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.476573 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.476789 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.477131 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260929 18:35:37.480103 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.482061 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.471524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56246)
0929 18:35:37.471703 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.471720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.471836 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.472588 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.472595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.472798 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.472955 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.472965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.474269 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.474272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.474919 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.474933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.475827 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.475832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.475941 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.476449 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.476462 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.477884 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.480038 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.480041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.480043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.480331 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.480938 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.480942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.480944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.481038 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.481571 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.481573 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.481661 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.481752 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.481864 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":3}
I20260929 18:35:37.482196 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.471675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.98.61:33177)
0929 18:35:37.472087 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.472100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.472124 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:35:37.472189 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.472195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.472409 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.472579 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.473104 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.474147 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.475065 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.475700 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.475820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.478031 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.478039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.478050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.478074 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.479909 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.480743 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.480754 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.480757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.480852 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.481139 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.481142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.481144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.481436 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.481566 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.481861 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.482028 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":104,"threads_started":1}
I20260929 18:35:37.483908 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.484654 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.484370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60200)
0929 18:35:37.484506 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.484574 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:35:37.485565 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.485242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60210)
0929 18:35:37.485425 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.485502 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 18:35:37.485797 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 18:35:37.490425 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.480649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56252)
0929 18:35:37.480733 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.480747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.480827 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.481384 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.481387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.481557 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.481666 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.481671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.482843 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.482846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.483588 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.483597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.485172 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.485179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.485279 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.485785 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.485801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.487091 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.488889 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.488891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.488893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.489138 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.489430 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.489432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.489433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.489515 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.489899 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.489901 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.489982 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.490117 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.490255 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 18:35:37.490593 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.480719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.98.61:33177)
0929 18:35:37.481069 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.481075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.481084 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:35:37.481133 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.481136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.481176 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.481765 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.481835 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.482711 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.483708 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.485001 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.485126 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.487226 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.487229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.487231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.487251 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.488787 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.489218 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.489220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.489222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.489293 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.489637 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.489639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.489640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.489797 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.489888 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.490191 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.490332 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":96,"threads_started":1}
I20260929 18:35:37.491333 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.491443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6eIoKd/build/debug/bin/kudu
/tmp/dist-test-task6eIoKd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38559
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:37.491983 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.491754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60226)
0929 18:35:37.491870 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.491935 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:35:37.492247 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260929 18:35:37.493219 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.493021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60240)
0929 18:35:37.493125 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.493170 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:35:37.493371 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260929 18:35:37.516959 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 1
I20260929 18:35:37.517809 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.517516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60244)
0929 18:35:37.517667 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.517728 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:35:37.518606 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.518395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60256)
0929 18:35:37.518507 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.518551 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:35:37.518904 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260929 18:35:37.542639 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 1
I20260929 18:35:37.543525 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.543220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60266)
0929 18:35:37.543386 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.543448 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 18:35:37.544337 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.544075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60278)
0929 18:35:37.544235 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.544281 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:35:37.544469 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260929 18:35:37.594117 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 2
I20260929 18:35:37.594947 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.594653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60292)
0929 18:35:37.594802 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.594868 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:35:37.595664 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.595451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60296)
0929 18:35:37.595559 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.595610 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:35:37.595798 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
W20260929 18:35:37.602725 23170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:35:37.603030 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:37.603091 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:37.603168 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:37.607291 23170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:37.607358 23170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:37.607402 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:37.607445 23170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:37.607501 23170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:37.612623 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 2
I20260929 18:35:37.613476 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.613169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60306)
0929 18:35:37.613300 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.613365 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:35:37.613655 23170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32877,127.0.98.61:33177,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:54:46 UTC on ccycloud.quasar-azztjd.root.comops.site
build id 4680912
I20260929 18:35:37.614162 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.613949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60320)
0929 18:35:37.614057 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.614108 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:35:37.614307 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260929 18:35:37.615042 23170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:37.616045 23170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:37.624759 23176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:37.624782 23178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:37.624730 23175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:37.625135 23170 server_base.cc:1034] running on GCE node
I20260929 18:35:37.625641 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:37.626717 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:37.627875 23170 hybrid_clock.cc:648] HybridClock initialized: now 1790706937627850 us; error 40 us; skew 500 ppm
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:37.630702 23170 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 18:35:37.631606 23170 webserver.cc:460] Webserver started at http://127.0.98.60:46525/ using document root <none> and password file <none>
I20260929 18:35:37.632247 23170 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:37.632313 23170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:37.632560 23170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:37.634598 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data/instance:
uuid: "0b98c27a9a4241a48cc87899faa493d5"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "fa07ab0546df264bbb15823d404ccd9d"
server_key_iv: "8138dfe9baf0d695b50c6d7ce4260b08"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.635147 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal/instance:
uuid: "0b98c27a9a4241a48cc87899faa493d5"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "fa07ab0546df264bbb15823d404ccd9d"
server_key_iv: "8138dfe9baf0d695b50c6d7ce4260b08"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.638868 23170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:37.641358 23185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:37.642547 23170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:35:37.642680 23170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal
uuid: "0b98c27a9a4241a48cc87899faa493d5"
format_stamp: "Formatted at 2026-09-29 18:35:37 on dist-test-slave-fmv8"
server_key: "fa07ab0546df264bbb15823d404ccd9d"
server_key_iv: "8138dfe9baf0d695b50c6d7ce4260b08"
server_key_version: "encryptionkey@0"
I20260929 18:35:37.642789 23170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:37.656944 23170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:37.659909 23170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:37.660113 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:37.666980 23170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38559
I20260929 18:35:37.667244 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38559 every 8 connection(s)
I20260929 18:35:37.667973 23170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/data/info.pb
I20260929 18:35:37.670784 23238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:37.674499 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.677954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6eIoKd/build/debug/bin/kudu as pid 23170
I20260929 18:35:37.678100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6eIoKd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706655775522-392-0/minicluster-data/master-2/wal/instance
I20260929 18:35:37.678390 392 external_mini_cluster.cc:1428] Setting key d02d812f6cf50c61913fa8176a66e7b7
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:37.688912 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.676481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.0.98.62:32877)
0929 18:35:37.676644 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.676649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.677565 (+ 916us) server_negotiation.cc:407] Connection header received
0929 18:35:37.678584 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.678588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.678644 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.678736 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.680290 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.680791 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.682039 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.682253 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.682360 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.685059 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.685062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.685064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.685088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.686613 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.687194 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.687198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.687199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.687282 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.687640 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.687643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.687660 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.687923 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.688005 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.688596 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.688727 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:35:37.689812 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.679101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.0.98.60:38559)
0929 18:35:37.679440 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.679451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.679475 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:35:37.679547 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.679552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.679732 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.679897 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.680639 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.681489 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.682359 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.682705 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.682820 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.685415 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.685420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.685430 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.685450 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.687045 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.687522 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.687527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.687529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.687604 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.687864 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.687866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.687867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.688338 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.688575 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.688771 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.688883 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":4736,"thread_start_us":157,"threads_started":1}
I20260929 18:35:37.689812 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.676878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32877 (local address 127.0.0.1:44304)
0929 18:35:37.677426 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.677468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.678356 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.678954 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.678964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.679528 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.680104 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.680117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.680916 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.680921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.681920 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.681929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.682398 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.682403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.682566 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.683170 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.683195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.684849 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.686724 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.686734 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.686744 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.687100 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.687371 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.687374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.687376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.687544 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.688025 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.688031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.688338 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.688510 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.688813 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":183,"spinlock_wait_cycles":3328,"thread_start_us":150,"threads_started":1}
I20260929 18:35:37.692190 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.694953 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 3
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:37.700963 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.692677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33177 (local address 127.0.0.1:56260)
0929 18:35:37.692819 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.692831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.692891 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.693120 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.693126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.693303 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.693439 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.693445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.694167 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.694173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.694696 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.694704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.695256 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.695259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.695359 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.695808 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.695827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.697279 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.699296 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.699301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.699303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.699528 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.699825 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.699830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.699832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.699936 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.700453 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.700459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.700562 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.700675 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.700800 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 18:35:37.701066 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.692757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.61:33177)
0929 18:35:37.692879 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.692885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.692902 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:37.692977 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.692979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.693015 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.693080 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.693552 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.694058 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.694814 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.695146 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.695248 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.697464 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.697471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.697473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.697495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.699200 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.699639 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.699643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.699645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.699733 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.700044 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.700048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.700050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.700307 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.700404 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.700743 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.700879 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:37.701740 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:37.704375 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.692294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.0.98.62:32877)
0929 18:35:37.692417 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.692421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.692523 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:35:37.692687 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.692691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.692739 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.692791 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.693669 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.694146 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.694880 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.695428 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.695552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.699364 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.699368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.699370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.699394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.702520 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.702982 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.702985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.702987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.703087 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.703383 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.703388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.703390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.703620 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.703864 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.704069 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.704200 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:35:37.705366 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.695311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60328)
0929 18:35:37.695429 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.695445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.695521 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.695869 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.695872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.696060 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.696200 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.696208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.697196 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.697201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.698096 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.698110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.698743 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.698748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.698855 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.699291 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.699310 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.700713 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.703511 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.703514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.703516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.703744 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.704059 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.704063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.704064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.704144 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.704870 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.704873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.704968 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.705089 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.705219 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:35:37.705397 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.695348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.0.98.60:38559)
0929 18:35:37.695654 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.695656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.695665 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:35:37.695711 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.695717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.695755 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.695833 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.696324 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.697089 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.698235 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.698615 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.698734 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.700824 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.700830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.700836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.700861 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.703417 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.703848 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.703852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.703853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.703929 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.704410 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.704414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.704539 (+ 125us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.704735 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.704859 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.705152 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.705252 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260929 18:35:37.705518 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 3
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:37.709689 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: Bootstrap starting.
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 18:35:37.712218 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.702221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60342)
0929 18:35:37.702604 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.702618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.702676 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.702984 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.702988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.703186 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.703301 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.703308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.704793 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.704797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.705522 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.705532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.706221 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.706224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.706319 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.706754 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.706768 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.708117 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.710123 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.710126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.710128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.710952 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.711250 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.711253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.711254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.711330 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.711764 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.711766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.711834 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.711919 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.712046 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260929 18:35:37.713099 23087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: Neither blocks nor log segments found. Creating new log.
Sep 29 18:35:37 dist-test-slave-fmv8 krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 18:35:37.714112 23087 log.cc:824] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:37.714715 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.702355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.98.60:38559)
0929 18:35:37.702739 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.702748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.702760 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:35:37.702796 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.702801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.702847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.702946 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.703439 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.704699 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.705650 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.706020 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.706115 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.708279 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.708282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.708284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.708304 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.709992 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.711068 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.711071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.711073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.711156 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.711426 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.711428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.711430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.711649 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.711734 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.712451 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.712556 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":107,"threads_started":1}
I20260929 18:35:37.715950 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.706042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.98.60:38559)
0929 18:35:37.706263 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.706268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.706281 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:37.706313 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.706315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.706351 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.706756 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.706976 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.707603 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.709182 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.709474 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.709570 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.712525 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.712533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.712536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.712567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.714143 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.714707 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.714710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.714711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.714825 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.715164 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.715167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.715169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.715338 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.715446 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.715705 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.715812 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260929 18:35:37.715951 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.705843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:60354)
0929 18:35:37.706074 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:37.706090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:37.706166 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:37.706485 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:37.706488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:37.706660 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:37.706875 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.706886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.707732 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.707735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.709040 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:37.709050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.709676 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.709680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:37.709766 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.710181 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:37.710194 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:37.711928 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:37.714336 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.714340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.714344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.714602 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.714944 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:37.714949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:37.714954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:37.715053 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:37.715438 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:37.715441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:37.715538 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:35:37.715646 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:35:37.715785 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":487}
I20260929 18:35:37.717585 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.705645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.98.61:33177)
0929 18:35:37.705766 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.705772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.706168 (+ 396us) server_negotiation.cc:407] Connection header received
0929 18:35:37.706329 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.706333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.706378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.706433 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.707783 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.708901 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.710638 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.710986 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.711107 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.714019 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.714024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.714027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.714053 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.715714 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.716396 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.716399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.716401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.716478 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.716770 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.716773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.716775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.716955 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.717159 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.717337 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.717445 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260929 18:35:37.717967 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: No bootstrap required, opened a new log
I20260929 18:35:37.719051 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: Bootstrap starting.
I20260929 18:35:37.720629 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: Bootstrap starting.
I20260929 18:35:37.721055 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:37.722174 23061 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8a3b87fc85a54521aeae9a56c7ce341f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44314
I20260929 18:35:37.722126 23087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.722361 23087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3b87fc85a54521aeae9a56c7ce341f, State: Initialized, Role: FOLLOWER
I20260929 18:35:37.722642 23238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:37.722698 23087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.723061 23162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:37.723500 23238 log.cc:824] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:37.723838 23162 log.cc:824] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:37.724107 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.724220 23087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:35:37.724261 23040 tablet.cc:2369] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:37.724287 23061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:35:37.724409 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:37.724274 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.725581 23061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.726176 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: No bootstrap required, opened a new log
I20260929 18:35:37.726445 23061 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [CANDIDATE]: Term 1 election: Requested vote from peers 2a6a7299c06c43b48c6674f8b570e935 (127.0.98.61:33177), 0b98c27a9a4241a48cc87899faa493d5 (127.0.98.60:38559)
I20260929 18:35:37.728739 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: No bootstrap required, opened a new log
I20260929 18:35:37.730212 23162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.730443 23162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6a7299c06c43b48c6674f8b570e935, State: Initialized, Role: FOLLOWER
I20260929 18:35:37.730767 23162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.732208 23162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:37.732707 23238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.732913 23238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b98c27a9a4241a48cc87899faa493d5, State: Initialized, Role: FOLLOWER
I20260929 18:35:37.733196 23238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.730639 23019 master_runner.cc:186] Error getting master registration for 127.0.98.61:33177: OK, instance_id { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" instance_seqno: 1790706937450652 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:35:37.734387 23019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:37.734365 23238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:37.734408 23212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b98c27a9a4241a48cc87899faa493d5"
I20260929 18:35:37.734659 23212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:37.734632 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a6a7299c06c43b48c6674f8b570e935"
I20260929 18:35:37.735291 23136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:37.736057 23212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3b87fc85a54521aeae9a56c7ce341f in term 1.
I20260929 18:35:37.736596 23036 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b98c27a9a4241a48cc87899faa493d5, 8a3b87fc85a54521aeae9a56c7ce341f; no voters:
I20260929 18:35:37.736662 23136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3b87fc85a54521aeae9a56c7ce341f in term 1.
I20260929 18:35:37.736898 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:37.737095 23246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [term 1 LEADER]: Becoming Leader. State: Replica: 8a3b87fc85a54521aeae9a56c7ce341f, State: Running, Role: LEADER
I20260929 18:35:37.737401 23246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260929 18:35:37.739034 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3b87fc85a54521aeae9a56c7ce341f. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.739140 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:37.739279 23259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:37.739333 23259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:37.741760 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.741962 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:35:37.742170 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:37.742239 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:37.742491 23293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:37.745440 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.729883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.61:33177)
0929 18:35:37.729982 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.729987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.730845 (+ 858us) server_negotiation.cc:407] Connection header received
0929 18:35:37.731134 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.731137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.731182 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.731230 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.732795 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.733616 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.737142 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.737444 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.737532 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.739461 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.739465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.739467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.739492 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.742932 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.743447 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.743453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.743454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.743648 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.744123 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.744126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.744127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.744471 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.744643 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.745167 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.745287 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:35:37.745816 23293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 18:35:37.745896 23277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:37.745975 23277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:37.747452 23094 master_runner.cc:263] No leader master found from master 2a6a7299c06c43b48c6674f8b570e935
I20260929 18:35:37.747535 23094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:37.755059 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.735135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.0.98.62:32877)
0929 18:35:37.735425 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.735431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.746168 (+ 10737us) server_negotiation.cc:407] Connection header received
0929 18:35:37.746340 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.746343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.746387 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.746435 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.747350 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.748096 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.749223 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.749565 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.749643 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.751335 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.751339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.751341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.751370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.753164 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.753659 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.753662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.753664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.753749 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.754075 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.754078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.754080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.754293 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.754407 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.754783 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.754897 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260929 18:35:37.755484 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.755697 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.762290 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:37.741697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.98.60:38559)
0929 18:35:37.741799 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.741803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.752409 (+ 10606us) server_negotiation.cc:407] Connection header received
0929 18:35:37.752598 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.752601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.752653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.752704 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.753639 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.754479 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.756412 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:37.756783 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:37.756861 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:37.758648 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:37.758652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:37.758654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:37.758681 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:37.760462 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.760958 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.760962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.760964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.761045 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:37.761344 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:37.761347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:37.761349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:37.761575 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:37.761686 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:37.762024 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 18:35:37.762139 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:35:37.766636 23212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [term 1 FOLLOWER]: Refusing update from remote peer 8a3b87fc85a54521aeae9a56c7ce341f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:37.767457 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:37.772119 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3b87fc85a54521aeae9a56c7ce341f. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.772259 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.773329 23136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [term 1 FOLLOWER]: Refusing update from remote peer 8a3b87fc85a54521aeae9a56c7ce341f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:37.774510 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:37.778944 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3b87fc85a54521aeae9a56c7ce341f. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.779065 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.782601 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.782729 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:37.783864 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.783984 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.784444 23297 mvcc.cc:205] Tried to move back new op lower bound from 7334735617083731968 to 7334735616975421440. Current Snapshot: MvccSnapshot[applied={T|T < 7334735617083731968}]
I20260929 18:35:37.787498 23298 mvcc.cc:205] Tried to move back new op lower bound from 7334735617083731968 to 7334735616975421440. Current Snapshot: MvccSnapshot[applied={T|T < 7334735617083731968}]
I20260929 18:35:37.789791 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.789916 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:37.791026 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3b87fc85a54521aeae9a56c7ce341f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32877 } } peers { permanent_uuid: "2a6a7299c06c43b48c6674f8b570e935" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33177 } } peers { permanent_uuid: "0b98c27a9a4241a48cc87899faa493d5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260929 18:35:37.791131 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:37.791287 23293 catalog_manager.cc:1347] Generated new cluster ID: 8afd909e1af242a182413e63d6124630
I20260929 18:35:37.791332 23293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:37.830576 23293 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:37.831920 23293 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:37.848908 23293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3b87fc85a54521aeae9a56c7ce341f: Generated new TSK 0
I20260929 18:35:37.852432 23293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:35:37.892035 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.889544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.0.98.62:32877)
0929 18:35:37.889672 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.889677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.890933 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 18:35:37.891151 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.891155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.891207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.891261 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.891894 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44330: BlockingRecv error: recv got EOF from 127.0.0.1:44330 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:35:37.893587 23243 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.890350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.98.60:38559)
0929 18:35:37.890460 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.890464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.892712 (+ 2248us) server_negotiation.cc:407] Connection header received
0929 18:35:37.892888 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.892891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.892940 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.892991 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.893480 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60366: BlockingRecv error: recv got EOF from 127.0.0.1:60366 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:35:37.895433 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.894476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.98.61:33177)
0929 18:35:37.894610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.894615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.894634 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:35:37.894712 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.894715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.894762 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.894868 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.895318 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56280: BlockingRecv error: recv got EOF from 127.0.0.1:56280 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 18:35:37.899513 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.898024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.0.98.61:33177)
0929 18:35:37.898130 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.898134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.898581 (+ 447us) server_negotiation.cc:407] Connection header received
0929 18:35:37.898795 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.898798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.898871 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.898925 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.899413 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56296: BlockingRecv error: recv got EOF from 127.0.0.1:56296 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:35:37.905229 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.903734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.98.61:33177)
0929 18:35:37.903845 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.903849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.904346 (+ 497us) server_negotiation.cc:407] Connection header received
0929 18:35:37.904528 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.904532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.904581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.904631 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.905126 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56304: BlockingRecv error: recv got EOF from 127.0.0.1:56304 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 18:35:37.910977 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.909525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.61:33177)
0929 18:35:37.909628 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.909632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.910094 (+ 462us) server_negotiation.cc:407] Connection header received
0929 18:35:37.910291 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.910295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.910343 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.910392 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.910877 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56312: BlockingRecv error: recv got EOF from 127.0.0.1:56312 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 18:35:37.922072 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.920335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.0.98.61:33177)
0929 18:35:37.920492 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.920497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.921045 (+ 548us) server_negotiation.cc:407] Connection header received
0929 18:35:37.921275 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.921279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.921335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.921392 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.921940 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56318: BlockingRecv error: recv got EOF from 127.0.0.1:56318 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 18:35:37.934844 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.932836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.0.98.61:33177)
0929 18:35:37.932963 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.932968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.933709 (+ 741us) server_negotiation.cc:407] Connection header received
0929 18:35:37.933955 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.933960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.934024 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.934085 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.934708 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56326: BlockingRecv error: recv got EOF from 127.0.0.1:56326 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:35:37.955554 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.953835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.98.61:33177)
0929 18:35:37.953959 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.953963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.954499 (+ 536us) server_negotiation.cc:407] Connection header received
0929 18:35:37.954720 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.954723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.954778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.954830 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.955432 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56342: BlockingRecv error: recv got EOF from 127.0.0.1:56342 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:35:37.991567 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:37.990399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.0.98.61:33177)
0929 18:35:37.990583 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:37.990587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:37.990658 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:35:37.990826 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:37.990830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:37.990885 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:37.990974 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:37.991465 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56344: BlockingRecv error: recv got EOF from 127.0.0.1:56344 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 18:35:38.059018 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:38.057903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.98.61:33177)
0929 18:35:38.058085 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.058089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.058112 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:35:38.058257 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.058259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.058307 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.058379 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:38.058874 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56350: BlockingRecv error: recv got EOF from 127.0.0.1:56350 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 18:35:38.190441 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:38.189175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.61:33177)
0929 18:35:38.189341 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.189345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.189480 (+ 135us) server_negotiation.cc:407] Connection header received
0929 18:35:38.189661 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.189664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.189750 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.189851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:38.190319 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56366: BlockingRecv error: recv got EOF from 127.0.0.1:56366 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 18:35:38.449883 23168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:38.448693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.0.98.61:33177)
0929 18:35:38.448833 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.448838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.448977 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:35:38.449150 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.449154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.449211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.449301 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:38.449771 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56376: BlockingRecv error: recv got EOF from 127.0.0.1:56376 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:35:38.743479 23288 catalog_manager.cc:1259] Loaded cluster ID: 8afd909e1af242a182413e63d6124630
I20260929 18:35:38.743564 23288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: loading cluster ID for follower catalog manager: success
I20260929 18:35:38.745267 23288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: acquiring CA information for follower catalog manager: success
I20260929 18:35:38.746479 23288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b98c27a9a4241a48cc87899faa493d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:38.747184 23277 catalog_manager.cc:1259] Loaded cluster ID: 8afd909e1af242a182413e63d6124630
I20260929 18:35:38.747267 23277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: loading cluster ID for follower catalog manager: success
I20260929 18:35:38.748802 23277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: acquiring CA information for follower catalog manager: success
I20260929 18:35:38.749876 23277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a6a7299c06c43b48c6674f8b570e935: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:38.968122 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:38.964208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.0.98.61:33177)
0929 18:35:38.964501 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.964505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.964521 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:35:38.964574 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.964576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.964712 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.964811 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:38.965139 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.965676 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.966581 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.966813 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.966903 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:38.967235 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:38.967680 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:38.967775 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:38.967906 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 18:35:38.967999 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":99,"threads_started":1}
I20260929 18:35:38.975430 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:38.971581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.98.60:38559)
0929 18:35:38.971834 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.971838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.971852 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:35:38.971995 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.971998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.972129 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.972294 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:38.972522 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.973125 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.973963 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.974202 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.974305 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:38.974622 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:38.975048 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:38.975184 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:38.975222 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:35:38.975274 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260929 18:35:38.982430 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:38.978739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.60:38559)
0929 18:35:38.978858 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.978861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.979145 (+ 284us) server_negotiation.cc:407] Connection header received
0929 18:35:38.979248 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.979251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.979301 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.979379 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:38.979689 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.980445 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.981302 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.981648 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.981745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:38.982009 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:38.982092 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:38.982160 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:38.982249 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 18:35:38.982300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:35:38.982756 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:38.978452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.0.98.61:33177)
0929 18:35:38.978594 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.978601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.978844 (+ 243us) server_negotiation.cc:407] Connection header received
0929 18:35:38.979036 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.979040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.979101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.979179 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:38.979575 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.980422 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.981604 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.981941 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.982050 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:38.982277 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:38.982356 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:38.982405 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:38.982588 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:35:38.982642 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:35:38.983503 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:38.978701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.98.62:32877)
0929 18:35:38.978932 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:38.978936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:38.978947 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:35:38.979092 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:38.979095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:38.979246 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:38.979373 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:38.979637 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.980278 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.981663 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:38.982021 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:38.982141 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:38.982542 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:38.982883 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:38.982977 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:38.983334 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 18:35:38.983399 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260929 18:35:38.986615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 23019
I20260929 18:35:38.993748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 23094
I20260929 18:35:39.000034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6eIoKd/build/debug/bin/kudu with pid 23170
2026-09-29T18:35:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3191 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6294 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283218 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST